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ABSTRACT 


This  capstone  project  determines  whether  sensemaking  of  soldier  narratives  can  inform 
the  Department  of  Defense’s  (DOD)  capability  development  process  (CDP). 
Sensemaking  is  the  process  of  creating  awareness  and  understanding  in  situations  of  high 
complexity  or  uncertainty.  The  authors  gathered  service  member  narratives  concerning 
their  use  of  fielded  equipment,  which  created  metadata  for  both  quantitative  and 
qualitative  research  and  analysis.  This  capstone  compares  results  from  sensemaking  of 
narratives  with  results  from  the  Warfighter  Technology  Tradespace  Methodology 
(WTTM),  a  system  designed  for  the  rapid  fielding  of  equipment  for  small  forward 
operating  bases  (FOBs)  and  combat  outposts  (COPs).  The  capstone  finds  that  1)  soldier 
narratives  inform  the  fielding  process  by  providing  an  additional  layer  of  meaning  and 
context,  and  2)  soldier  narratives  do  not  replace  current  feedback  mechanisms;  rather, 
they  play  a  complementary  role.  This  capstone  finds  that  narratives  as  a  feedback 
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I.  INTRODUCTION 


A.  BACKGROUND 

Sensemaking  of  narratives  as  a  feedback  mechanism  is  rich  with  information  and 
context  that  has  the  potential  to  advance  the  current  development  and  fielding  process  of 
equipment.  Historically,  the  feedback  mechanism  for  developing  and  fielding  equipment 
has  been  problematic  in  the  United  States  (U.S.)  Army.  In  the  late  1800s,  the  U.S.  Army 
employed  special  boards  to  evaluate,  recommend,  and  field  equipment  on  a  large  scale 
from  clothing  to  arms,i  which  would  eventually  evolve  into  the  capabilities  development 
process  (CDP).2  Despite  the  CDP  becoming  the  template  for  developing  new  equipment, 
not  all  equipment  was  suitable.  For  example,  after  the  American  Civil  War  between  1870 
and  1880,  footgear  for  infantry  soldiers  was  poorly  engineered  yet  still  fielded.  3 

The  feedback  mechanism  still  experiences  flaws  in  some  of  today’s  military 
equipment  development  and  fielding  process.  In  1995,  Douglas  C.  McChristian  wrote, 
“The  army  struggled  within  its  own  bureaucracy  to  provide  what  it  deemed  best  for  its 
troops,  whereas  the  soldiers  themselves,  seeking  more  practical  solutions,  exercised  their 
initiative  by  obtaining  what  the  army  either  over-looked  or  refused  to  issue.”^  A 
disconnect  in  information  feedback  exists  between  the  U.S.  Army  decision  makers  and 
soldiers  in  regard  to  fielded  equipment;  the  CDP  is  not  maximizing  the  use  of  soldiers’ 
experiences  expressed  as  narratives^  to  gain  useful  information.  McChristian ’s 
observations  are  still  relevant. 


^  Douglas  C.  McChristian,  The  U.S.  Army  in  the  West,  1870-1880:  Uniforms,  Weapons,  and 
Equipment  (Norman,  OK:  University  of  Oklahoma  Press,  2006). 

3  The  capabilities  development  process  is  the  process  for  developing,  testing,  and  fielding  the  most 
reliable  and  effective  piece  of  equipment  to  enhance  mission  success.  Defense  Acquisition  University, 
“Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System,”  June  15, 
2010,  https://ilc.dau.mil/pdf/ILC_hotspots_linked_pdf.pdf. 

3  McChristian,  The  U.S.  Army  in  the  West,  1870-1880. 

4  Ibid. 

3  Manfred  Jahn  defines  narratives  as  “anything  that  tells  or  presents  a  story,  be  it  by  text,  picture, 
performance,  or  a  combination  of  these.”  Manfred  Jahn,  “Narratology:  A  Guide  to  the  Theory  of 
Narrative,”  Narratology:  A  Guide  to  the  Theory  of  Narrative,  May  28,  2005,  http://www.uni- 
koeln .  de/~  ame02/pppn .  htm#N  2 . 
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Although  soldiers  use  narratives  to  share  their  experiences  about  problematic 
fielded  equipment,  such  as  the  Army  Combat  Uniform  (ACU),  and  the  Land  Warrior 
System  (LWS),  U.S.  Army  decision  makers  have  not  effectively  analyzed  these 
narratives  to  capture  their  rich  information.  The  ACU,  developed  in  2003,  and  fielded  in 
2005,  provided  inadequate  camouflage  pattern  for  most  combat  environment.  6  In  2007, 
Eric  Coulson,  a  U.S.  Army  officer,  stated,  “Considering  all  the  testing  the  uniform  went 
through,  it  is  surprising  such  a  mediocre  product  finally  emerged... and  I  am  not  alone. 
I’ve  talked  to  many  soldiers  that  have  had  this  happen.”^  Four  years  after  fielding,  the 
U.S.  Congress  finally  acknowledged  the  issues  with  the  ACU  and  directed  the 
development  and  fielding  of  the  Operation  Enduring  Freedom  camouflage  pattern  (OCP) 
in  2009.^  However,  this  uniform  only  applied  to  units  deploying  to  Afghanistan,  which 
did  not  fix  the  ACU  issue  for  the  entire  U.S.  Army.  As  of  2014,  no  official  decision  has 
been  made  to  replace  the  ACU.^ 

In  1994,  the  U.S.  Army  began  developing  the  EWS.i®  The  EWS  was  a  digital 
integrated  fighting  system  for  individual  infantry  soldiers  intended  to  give  them  enhanced 
tactical  awareness,  lethality,  and  survivability.  1 1  Soldiers  stated  that  the  EWS  did  not 
perform  these  functions.  In  2005,  a  Department  of  Defense  (DOD)  report  showed  that 
soldiers  criticized  the  EWS  for  “excessive  weight,  poor  communication  and  a  number  of 
human  factors  concerns. In  2009,  Sergeant  James  Young  stated  that  the  EWS  was  “a 

^  GAO.gov,  “Warfighter  Support:  DOD  Should  Improve  Development  of  Camouflage  Uniforms  and 
Enhance  Collaboration  Among  the  Services,”  September  28,  2012,  http://www.gao.gov/products/GAO-12- 
707. 

^  Eric  Coulson,  “New  Army  Uniform  Doesn’t  Measure  Up,”  Military.com,  April  5,  2007,  http://www. 
military.com/NewsContent/0, 133 19,131 103, 00. html. 

^  GAO.gov,  “Warfighter  Support:  DOD  Should  Improve  Development  of  Camouflage  Uniforms  and 
Enhance  Collaboration  Among  the  Services.” 

9  Washington  Post,  “The  U.S.  Military’s  Changing  Camouflage,”  accessed  February  21,  2014, 
http://apps.washingtonpost.eom/g/page/politics/the-us-militarys-changing-camouflage/140/. 

Army-Technology.com,  “Land  Warrior  Integrated  Soldier  System — Army  Technology,”  accessed 
February  21,  2014,  http://www.army-technology.com/projects/land_warrior/. 

Army-Technology.com,  “Land  Warrior  Integrated  Soldier  System — Army  Technology.”  The 
systems  integrated  into  Land  Warrior  are  the  weapon  system,  helmet,  computer,  digital  and  voice 
communications,  positional  and  navigation  system,  protective  clothing  and  individual  equipment. 

Sandra  Erwin,  “Army  ‘Land  Warrior’  Not  Yet  Ready  for  War,  Gears  Up  for  Next  Tryout,”  NDIA, 
March  2006,  http://www.nationaldefensemagazine.org/archive/2006/March/Pages/land_warrior3037.aspx. 
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bunch  of  stuff  we  don’t  use,  taking  the  place  of  useful  stuff  like  guns... It  makes  you  a 
slower,  heavier  target.”i3  Based  on  soldier  narratives,  the  U.S.  Army  unsuccessfully 
attempted  to  trim  down  the  system,  eventually  canceling  the  program  a  decade  later, 
without  it  ever  being  fielded. 

These  two  examples  show  that  narratives  can  provide  powerful  information. 
However,  decision  makers  need  to  understand  how  to  make  sense  of  the  narrative  to 
access  such  information.  One  process  for  gathering  and  analyzing  this  information  is 
known  as  sensemaking.  Sensemaking  is  the  process  of  creating  awareness  and 
understanding  in  situations  of  high  complexity  or  uncertainty  to  make  decisions. 

B.  PURPOSE  AND  SCOPE 

Research  has  identified  that  a  relationship  exists  between  sensemaking  and  the 
ability  to  understand  and  derive  meaning  from  narratives.  The  purpose  of  this  research 
is  to  determine  whether  sensemaking  of  narratives  can  inform  the  Army  CDP. 
Specifically,  the  scope  of  this  research  analyzes  narratives  using  the  sensemaking  process 
to  inform  development  and  fielding  decisions.  This  research  refers  to  the  sensemaking 
process  and  the  narrative  in  a  broad  sense  to  understand  information.!^  Narratives  and 
stories  are  terms  used  synonymously  throughout  this  study.  This  research  does  not 
analyze  how  individuals  use  sensemaking  to  create  narratives  from  experiences. 
Furthermore,  this  research  compares  information  gained  on  the  ground-based  observation 
and  surveillance  system  (GBOSS)  using  two  different  assessment  methods. 


Noah  Shachtman,  “The  Army’s  New  Land  Warrior  Gear:  Why  Soldiers  Don’t  Like  It,”  Popular 
Mechanics,  October  1,  2009,  http://www.popularmechanics.com/technology/military/4215715. 

Defense  Industry  Daily,  “Slimmer,  Trimmer  U.S.  Army  Land  Warrior  System  Moves  Ahead,” 
Defense  Industry  Daily,  September  26,  2010,  http://www.defenseindustrydaily.com/Slimmer-Trimmer-US- 
Army-Land-Warrior-System-Moves-Ahead-05890/. 

David  Snowden,  “What  Is  Sense-Making? — Cognitive  Edge  Network  Blog,”  Cognitive  Edge 
Network,  June  7,  2008,  http://cognitive-edge.com/blog/entry/3840/what-is-sense-making/. 

Andrew  D.  Brown,  Patrick  Stacey,  and  Joe  Nandhakumar,  “Making  Sense  of  Sensemaking 
Narratives,”  Human  Relations  61,  no.  8  (2008):  1035-62. 

Zhang  et  al.  refers  to  broad  sense  of  sensemaking  as  the  total  process  of  (1)  searching  for 
information  and  (2)  making  sense  of  information.  Pengyi  Zhang  et  al.,  “Extending  Sense-Making  Models 
with  Ideas  from  Cognition  and  Learning  Theories,”  Proceedings  of  the  American  Society  for  Information 
Science  and  Technology  45,  no.  1  (2008):  23-23. 
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C.  SUMMARY  OF  FINDINGS 

This  research  determined  that  sensemaking  of  narratives  can  inform  the  CDP. 
Analysis  of  service  member  narratives  provided  similar  qualitative  information  that 
aligned  with  current  feedback  methodologies  assessing  fielded  equipment.  Service 
member  narratives  do  not  replace  current  feedback  mechanisms,  but  rather  they  play 
complementary  role. 

D.  CHAPTER  OVERVIEW 

Chapter  II  describes  sensemaking  and  narrative  theories  in  the  context  that  they 
apply  to  this  research.  Chapter  III  illustrates  the  U.S.  Army  Capabilities  Development 
Process  (CDP)  and  the  Warfighter  Technology  Tradespace  Methodology  (WTTM). 
Chapter  IV  describes  the  methodology  of  the  research  with  descriptions  and  explanations 
of  the  research  test  subjects,  the  collection  tool  (SenseMaker  Collector),  and  the  analysis 
tool  (SenseMaker  Explorer).  Chapter  V  provides  the  research  findings  to  determine  if 
sensemaking  of  narratives  informs.  Finally,  Chapter  VI  explores  the  potential  use  of 
sensemaking  in  other  facets  of  the  DOD. 
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II.  THEORETICAL  ERAMEWORK 


This  chapter  introduces  a  researcher-developed  model  to  describe  the  relationship 
between  the  narrative  and  sensemaking  theories.  Additionally,  the  model  is  explained  in 
two  separate  processes  to  show  how  individual  experiences  are  transformed  into 
information. 

A.  CREATING  NARRATIVES  THROUGH  SENSEMAKING 

In  this  research,  sensemaking  is  used  in  two  contexts  in  relation  to  the  narrative. 
First,  individuals  make  sense  of  their  experiences  to  create  narratives;  second,  researchers 
then  make  sense  of  these  narratives  to  gain  information.  The  second  context  is  the  focus 
of  this  research.  This  relationship  is  depicted  in  the  sensemaking-experience-narrative- 
sensemaking-information  (SENSI)  model  shown  in  Figure  1. 


SENSEMAKING 

A 

^  NARRATIVE  — > 

SENSEMAKING 

Soldiers  making  sense  of 

V 

Making  sense  of  soldier 

the  their  experiences 

narratives  about  fielded 

about  fielded  equipment 

equipment 

INFORMATION 


Figure  1.  SENSI  model:  Making  sense  of  experiences  and  narratives 


Humans  communicate  through  oral  or  written  stories  to  influence,  capture  events, 
and  understand  and  express  simple  and  complex  ideas.  Storytelling  is  a  uniting  and 
defining  component  of  all  communities. Stories  have  empowered  social  movements  by 
providing  context  of  shared  experiences.  According  to  Polletta,  activists  in  social 
movements  use  storytelling  to  mobilize  people  and  gain  popular  support,  Human 

David  Snowden,  “Story  Telling:  An  Old  Skill  in  a  New  Context,”  Business  Information  Review  16, 
no.  1  (1999):  30-37. 

Francesca  Polletta,  “Culture  and  Movements,”  The  Annals  of  the  American  Academy  of  Political 
and  Social  Science  619,  no.  1  (2008):  78-96. 
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communication  through  storytelling  is  possible  because  of  common  languages,  which  are 
associated  to  meanings. 20 

The  transfer  of  knowledge  and  learning  is  the  fundamental  mechanism  of  the 
narrative. 21  Walter  R.  Fischer’s  narrative  paradigm  makes  it  possible  to  assume  that 
stories  are  the  basic  form  of  human  communication.  The  narrative  paradigm  states  that 
human  communication  should  be  viewed  as  historical,  as  well  as  situational,  as  stories 
competing  with  other  stories  constituted  by  good  reasons,  as  being  rational  when  they 
satisfy  the  demands  of  narrative  probability  and  narrative  fidelity,  and  as  moral 
inducements. 22  Fisher  claims  that  humans  communicate  through  storytelling,  and  that 
they  make  sense  of  their  experiences  through  personal  narratives.  Fisher’s  claim,  in  other 
words,  addresses  the  first  context  of  sensemaking  in  the  SENSI  model. 

Sensemaking  in  the  first  context  of  the  SENSI  model  is  a  method  in  which 
humans  integrate  experiences  into  distinct  stories  or  narratives.  Karl  Weick  states: 

The  role  of  stories  in  sensemaking  has  been  given  considerable  attention 
recently,  due  in  part  to  Mitroff  and  Kilmann’s  (1976)  pilot  study,  Eisher’s 
(1984)  systematic  thinking,  Polkinghorne’s  (1988)  survey,  and  Bruner’s 
(1990)  and  Zukier’s  (1986)  engaging  discussions  of  the  idea  that  people 
think  narratively  rather  than  argumentatively  or  paradigmatically.23 

Weick  highlights  the  relationship  and  importance  that  narratives  have  in 
sensemaking.  When  one  individual  or  a  group  of  people  shares  their  story  or  narrative, 
people  interprets  or  makes  sense  of  the  information  in  their  own  way  and  from  their  own 
viewpoint  and  familiarities.  To  understand  this  phenomenon  of  narratives,  an  entire 
discipline  was  created  in  academia  called  narratology. 


20  Dan  Sperber,  “How  Do  We  Communicate,”  How  Things  Are:  A  Science  Toolkit  for  the  Mind,  1995, 
191-99. 

21  Snowden,  “Story  Telling.” 

22  Walter  R.  Fisher,  “Narration  as  a  Human  Communication  Paradigm:  The  Case  of  Public  Moral 
Argument,”  Communications  Monographs  51,  no.  1  (1984):  1-22. 

23  Karl  E.  Weick,  Kathleen  M.  Sutcliffe,  and  David  Obstfeld,  “Organizing  and  the  Process  of 
Sensemaking,”  Organization  Science  16,  no.  4  (2005):  409-21. 
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Narrative  theory  is  a  humanities  discipline  dedicated  to  the  study  of  the  logic, 
principles,  and  practices  of  narrative  representation. 24  Tzvetan  Todorov  first  formally 
introduced  this  discipline  in  1969  by  proposing  “a  new  type  of  generalizing  theory  that 
could  be  applied  to  all  domains  of  narrative. ”25  However,  Todorov’s  initial  definition  of 
narratology  has  evolved  over  time  as  captured  by  Jan  Meister.  Meister  states: 

During  its  initial  or  “classical”  phase,  from  the  mid-1960s  to  the  early 
1980s,  narratologists  were  particularly  interested  in  identifying  and 
defining  narrative  universals.  This  tendency  is  still  echoed  in  a  concise 
1993  definition  of  narratology  as  “the  set  of  general  statements  on 
narrative  genres,  on  the  systematics  of  narrating  (telling  a  story)  and  on 
the  structure  of  plot”  (Ryan  &  von  Alphen  1993:  1 10). 

However,  a  decade  later,  narratology  was  alternatively  described  as  (a)  “a  theory  (Prince 
2003,  1),  (b)  a  method  (Kindt  &  Muller  2003,  211),  or  (c)  a  discipline  (Fludernik  & 
Margolin  2004,  149).”26 

All  three  definitions  of  narratology  from  the  mid-1960s  to  the  present  are  based 
upon  the  same  principle,  narratives  identify  meaning  through  experiences.  Narratives 
provide  meaning  across  a  wide  range  of  disciplines,  particularly  from  the  fields  of  social 
sciences,  such  as  anthropology  and  folklore,  psychology,  sociology,  law,  medicine, 
nursing,  and  social  work.22  Narratives  are  powerful  tools  to  assist  in  decision  making 
because  they  provide  information  and  context  to  problems  or  situations. 

B.  UNDERSTANDING  NARRATIVES  THROUGH  SENSEMAKING 

How  can  narratives  be  captured  and  analyzed  to  understand  their  meaning  to  gain 
information?  Sensemaking  makes  it  possible  to  take  a  complex  problem  or  idea  and 


24  Inderjeet  Mani,  “Revision  of  Computational  Narratology  from  Wed,  26.  June  2013,”  The  Living 
Handbook  of  Narratology,  January  28,  2013,  http://www.lhn.uni-hamburg.de/node/43/revisions/128/view. 

25  Christopher  Meister,  “Narratology,”  The  Living  Handbook  of  Narratology,  August  26,  2011, 
http://www.lhn.uni-hamburg.de/article/narratology. 

26  Ibid.  Jan  Meister  is  a  professor  of  Narratology  at  the  University  of  Hamburg,  Germany. 

22  Lewis  P.  Hinchman  and  Sandra  K.  Hinchman,  Memory,  Identity,  Community:  The  Idea  of  Narrative 
in  the  Human  Sciences  (Albany,  NY:  SUNY  Press,  1997). 
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understand  its  meaning  in  a  simplified  way  that  is  understandable  and  makes  sense.  ^8 
This  concept  is  the  second  context  of  sensemaking  in  the  SENSI  model.  This  research 
uses  sensemaking  to  deconstruct  narratives  through  pattern  and  narrative  analyses. 

Pattern  analysis  evaluates  information  through  conceptual  grouping,  organized  by 
themes,  to  gain  an  in-depth  understanding  of  the  experience  of  particular  individuals. 
Narrative  analysis  ranges  from  analyzing  verbal  and  non-verbal  quantitative  or  qualitative 
data.2^  Sensemaking,  through  pattern  and  narrative  analyses,  has  the  ability  to  extract 
information  embedded  within  narratives.  This  information  is  predominantly  qualitative 
and  provides  access  to  subjective  experiences  and  insights.  The  extracted  information 
from  the  narratives  is  the  final  product  of  the  SENSI  model. 

This  research  acknowledges  that  the  sensemaking  process  has  been  practiced  in 
many  disciplines  and  fields  of  study.  Several  examples  are  in  the  fields  of 
communications, 30  business, 3 1  social  sciences, 32  and  the  military,  specifically  in 
command  and  control  (C2),33  decision  making,34  and  intelligence  analysis. 35 

Sensemaking  of  narratives  is  a  powerful  tool  to  understand  experiences  and  ideas, 
which  provides  contextual  meaning  and  insight.  Narrative  and  sensemaking  theories  have 


28  Gary  Klein,  Brian  M.  Moon,  and  Robert  R.  Hoffman,  “Making  Sense  of  Sensemaking  1 :  Alternative 
Perspectives,”  IEEE  Intelligent  Systems  21,  no.  4  (2006):  70-73. 

29  Catherine  Kohler  Riessman,  “1  Narrative  Analysis,”  2005,  http://cmsu2.ucmo.edu/public/classes/ 
Baker%20COMM%205820/narrative%20analysis.riessman.pdf. 

30  Alan  H.  Schoenfeld,  “Learning  to  Think  Mathematically:  Problem  Solving,  Metacognition,  and 
Sense  Making  in  Mathematics,”  Handbook  of  Research  on  Mathematics  Teaching  and  Learning,  1992, 
334-70. 

31  Arch  G.  Woodside,  “Sensemaking  about  Business-to-Business  Strategies  and  Relationships:  A 
Commentary  on  Reid  and  Plank’s  Review,”  Journal  of  Business-to-Business  Marketing  7,  no.  4  (2000): 
45-53. 

32  Joyce  S.  Osland  and  Allan  Bird,  “Beyond  Sophisticated  Stereotyping:  Cultural  Sensemaking  in 
Context,”  The  Academy  of  Management  Executive  14,  no.  1  (2000):  65-77. 

33  Eva  Jensen,  “Sensemaking  in  Military  Planning:  A  Methodological  Study  of  Command  Teams,” 
Cognition,  Technology  &  Work  11,  no.  2  (2009):  103-18. 

34  S.  G.  McIntyre,  Marlene  Gauvin,  and  Barbara  Waruszynski,  “Knowledge  Management  in  the 
Military  Context,”  Canadian  Military  Journal  4,  no.  1  (2003):  35^0. 

35  Peter  Pirolli  and  Stuart  Card,  “The  Sensemaking  Process  and  Leverage  Points  for  Analyst 
Technology  as  Identified  through  Cognitive  Task  Analysis,”  in  Proceedings  of  International  Conference  on 
Intelligence  Analysis,  vol.  5,  2005,  2— A. 
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been  successfully  applied  to  various  disciplines  and  have  the  potential  application  to  the 
Army’s  CDP.  The  ability  to  deconstruct  narratives  through  sensemaking  to  obtain  a 
deeper  understanding  could  have  the  potential  to  inform  current  and  future  equipment 
development  processes,  bridging  the  gap  between  decision  makers  and  warfighters. 
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III.  DEVELOPMENT  AND  ASSESSMENT  PROCESSES 


This  chapter  provides  an  overview  of  the  DOD’S  CDP  and  the  WTTM.  This 
research  also  identifies  possible  areas  within  these  two  processes  in  which  narratives 
could  be  employed  to  gain  service  member  feedback  on  fielded  equipment. 

A.  CAPABILITIES  DEVELOPMENT  PROCESS 

This  process  is  a  component  of  the  integrated  defense  acquisition,  technology,  and 
logistics  life  cycle  management  system  that  provides  service  members  with  the  most 
reliable  and  effective  piece  of  equipment  to  enhance  mission  success. 3^  The  CDP  is 
divided  into  five  phases:  material  solution  analysis  phase,  technology  development  phase, 
engineering  and  manufacturing  development  phase,  production  and  deployment  phase, 
and  operations  and  support  phase  (see  Appendix  A,  Integrated  Defense  Acquisition, 
Technology,  and  Logistics  Management  Life  Cycle). Although  specific  in  each 
function,  and  well  delineated,  phases  blend  together  to  maximize  integration  and 
momentum  in  transitioning  from  one  phase  to  the  other.  The  ultimate  goal  is  a  support 
program  that  meets  material  readiness  and  operational  support  performance  requirements 
and  sustains  a  system  in  most  cost-effective  manner.  38 

The  initiating  step  in  the  CDP  is  defining  the  customer’s  requirements  of  a  piece 
of  equipment.  Customers  range  from  Combatant  Command  hierarchies.  Army-level  staff 
initiatives,  DOD  programs.  Congressional  mandates  and  service  members.  Annotation 
and  delivery  of  a  customer’s  needs  or  requirements  is  executed  through  operational  need 
statements,  think  tanks,  or  higher  authority  directives.  This  process  creates  several  levels 
of  separation  from  the  originator  to  the  developer.  The  combat  developer  and  the  Army 


36  The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  provides  oversight  and 
management  of  the  CPD  through  each  service  component’s  Office  of  the  Assistant  Secretary  for 
Acquisition,  Logistics,  and  Technology.  OSD.ATL-Webmaster@mail.mil,  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics,  “Welcome  to  AT&L,”  accessed  March  31,  2014, 
http://www.acq.osd.mil/. 

3^  Defense  Acquisition  University,  “Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life 
Cycle  Management  System.” 

38  Ibid. 
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J3/5/7  Joint  Staff  (strategic  operations,  force,  and  policy  development)  must  then 
determine  the  functions  of  the  system  and  what  decision  makers  value  through 
stakeholder,  functional,  and  value  modeling  analysis.  Specifically,  they  define  and 
validate  the  requirement  in  the  form  of  Doctrine,  Organization,  Training,  Materiel, 
Leadership  and  Education,  Personnel,  and  Facilities  (DOTMLPF).39 

Once  the  DOTMFPF  is  revised  and  refined,  based  on  the  needs,  wants,  and 
desires  of  the  customer,  the  requirement  document  is  tasked  to  the  Acquisition  Corps, 
specifically,  a  material  developer  for  initiation  of  phase  one  of  the  CDP,  the  material 
solution  analysis  phase.  A  material  developer  must  dissect  the  requirement  to  allow  the 
ability  to  employ  the  most  subordinate  and  intermediate  cost  effective  material  solutions 
per  entity  of  the  requirement.  A  material  developer  has  the  option  to  select  commercial 
over  the  shelf  (COTS)  and  army  supply  system  materials  from  any  range  of  companies.  If 
unable  to  use  either  of  these  solutions,  or  only  able  to  utilize  in  support  of  select  entities 
of  the  requirement,  the  material  manager  can  build  a  technology  development  contract 
with  an  accredited  company. The  final  output  of  the  first  phase  is  a  product  support 
strategy  that  encompasses  a  fully  defined  requirement,  architectural  solution,  concept  of 
operational  equipment  development,  and  adequate  allocated  funds  in  support  of  further 
development.  As  previously  mentioned,  the  initiation  of  the  technology  development 
phase  begins  upon  finalization  of  the  architectural  solution,  but  phase  1  does  not  end  until 
all  required  materials  for  the  solution  are  accounted  for  physically  or  contractually. 

During  the  second  phase,  technology  development,  the  material  developer  works 
with  and  manages  the  contracted  companies  tasked  to  synchronize  materials  solutions 
with  technological  advances  and  develop  initial  prototypes.  Development  of  initial 
prototypes  provides  a  litmus  test  and  azimuth  check  for  the  material  and  combat 
developer,  as  well  as  for  army  staff  to  validate  that  the  pre-defined  architectural  solution 
and  concept  of  the  operational  equipment’s  development  are  in  line  with  the  customer 
requirements  and  production  rate.  The  initial  prototype  is  validated  through  testing 

39  Chairman  of  the  Joint  Chief  of  Staff,  Joint  Capabilities  Integration  and  Development  System,  Joint 
Chief  of  Staff  Instruction,  January  10,  2012. 

Defense  Acquisition  University,  “Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life 
Cycle  Management  System.” 


12 


functional  and  enabling  specifications,  as  thoroughly  defined  from  the  customer’s 
requirement.  The  validation  of  the  overall  system’s  performance  triggers  the  evolutionary 
acquisition  strategy,  which  includes  full  level  accreditation  of  the  strategy  by  the  Army 
staff  and  the  DOD.  The  prototype  is  then  transitioned  to  a  program  manager  (PM)  to 
proceed  with  increased  production  of  the  prototype  and  to  build  the  engineering  and 
manufacturing  contract,  which  is  the  start  of  the  engineering  and  manufacturing 
development  phase  and  end  of  phase  2.4i 

Within  the  third  phase,  engineering  and  manufacturing  development,  the  PM 
transitions  from  a  solicitor  and  manager  to  strictly  a  manager,  who  manages  the  prototype 
or  equipment  production  rate  throughout  the  longest  phase  of  the  CDP.  The  main  priority 
during  this  phase  is  meeting  the  initial  product  baseline  with  all  prototypes  in  compliance 
with  the  validated  performance  functions,  prescribed  system  threat  assessment,  and 
sustainment  management  requirement.  Once  the  evolution  of  a  product  baseline  is 
reached,  the  first  systems  engineering  test  and  validation  supportability  program,  also 
referred  to  as  the  development  test  (DT),  is  initiated  through  Department  of  the  Army 
accredited  test  boards,  civilian  organizations,  and  military  units.  The  goal  of  the  test 
program  is  to  use  the  equipment  in  scripted  simulated  military  situations  in  which  the 
equipment  will  ideally  be  utilized  in  the  future  in  support  of  assigned  missions,  combat  or 
training.  The  end  result  of  the  test  program  is  a  conceptual  and  thorough  report,  which 
states  that  the  prototype  or  equipment  meets  all  the  prescribed  functional  requirements 
and  specifications,  is  durable,  and  meets  the  safety  requirements  defined  by  the  DOD. 
The  positive  test  report  grants  the  PM  the  certification  to  submit  to  the  Army  Staff  and 
the  DOD  a  request  for  mass  production  of  the  equipment  (no  longer  a  prototype)  and 
initiate  the  final  evolutionary  acquisition  strategy. 

The  initial  manufacturing  rate  in  the  final  two  phases,  production  and  deployment 
as  well  as  operations  and  support,  begins  at  a  low  rate  baseline  to  allow  for  a  final  test 
phase  and  an  opportunity  for  final  adjustments  and  corrective  actions  to  provide  the  best 

Defense  Aequisition  University,  “Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life 
Cycle  Management  System.” 

42  Ibid. 
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material  solution  for  the  customer.  The  PM  can  conduct  an  operational  testing  (OT)  on 
the  equipment,  which  includes  a  non- scripted  test  by  an  operational  Army  unit  during 
training  exercises.  The  current  feedback  and  report  produced  from  this  test  program  is  not 
defined  in  a  prescribed  or  mandated  document;  it  is  simply  at  the  control  or  preference  of 
the  PM.  Upon  conclusion  of  the  final  test  program  and  address  of  any  adjustments  or 
modifications  needed,  the  PM  moves  to  final  validation  by  the  Army  J3/5/7  and  combat 
developer  to  gain  the  approval  to  initiate  the  fielding  phase.  The  initial  equipment 
fielding  phase  includes  a  small  baseline  of  the  equipment  fielded  to  selected  units  for 
testing  in  either  a  training  or  combat  environment.  During  this  phase,  the  soldiers  of  the 
receiving  units  participate  in  training  on  the  operational  employment  and  sustainment  of 
the  fielded  equipment,  and  may  be  permanently  tasked  a  field  service  representative 
(FSR)  for  depot  level  maintenance  or  technological  issues. 

Following  the  initial  fielding  phase,  the  PM  has  an  option  to  gain  feedback  from 
operational  units  or  assessment  organizations  on  the  performance,  durability,  and  overall 
function  of  the  equipment  to  acquire  qualitative  data  for  any  modifications  or 
adjustments.  Examples  of  assessment  organizations  are  the  Forward  Operational 
Assessment  teams  (FOA)44  and  the  Maneuver  Center  of  Excellence  (MCoE).  The  EOA’s 
feedback  mechanism  includes  surveys  and  interviews  to  gain  information  on  equipment 
being  used  in  operational  settings.  The  collected  soldier  feedbacks  are  used  to  generate  a 
report  called  the  forward  operational  assessment  report  for  Army  leadership  and 
capabilities  developers  to  inform  the  CDP.45  MCoE’s  feedback  mechanisms  include 
surveys  comprised  of  Eikert  Scales,  multiple  choices,  and  opened  ended  questions  to  ask 
soldiers  to  rate  the  effectiveness  or  importance  of  U.S.  Army  systems  (equipment  and 


43  John  Bryan,  PEO  soldier  interview,  October  28,  2013. 

44  The  Forward  Operational  Assessment  (FOA)  teams  are  subordinate  to  the  Operational  Testing 
Center  (OTC)  within  The  United  States  Army  Test  and  Evaluation  Command  (ATEC).  Shane  Dietrich, 
Wartime  Test  and  Evaluation;  Initiatives  Lead  to  Cultural  Change  (research  paper,  Carlisle  Barracks,  PA: 
U.S.  Army  War  College,  2007). 

45  David  Taylor,  “Forward  Operational  Assessment  Team  Survey  (Long  Version),”  DVIDS,  accessed 
November  21,  2013,  http://www.dvidshub.net/video/302960/forward -operational-assessment-team-survey- 
long-version. 
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training). 46  Similar  to  the  operational  testing  program,  utilizing  these  feedbaek 
meehanisms  are  neither  mandated  nor  even  exeeuted  by  all  PMs.  Onee  the  initial  fielding 
phase  is  complete,  and  the  identified  initial  baseline  of  equipment  is  fielded,  the  PM 
moves  to  high-rate  baseline  production  and  fielding. 

High-rate  fielding  is  the  final  step  in  the  CDP.  It  is  the  point  in  the  phase  at  which 
mass  production  of  the  equipment  is  complete  and  all  equipment  is  fielded  to  the  selected 
units  in  accordance  with  the  prescribed  army  mission  table  of  organization  and  equipment 
(MTOE).  If  any  additional  issues  or  requirements  surface  during  this  stage,  the  PM  has 
limited  options  to  address  these  issues  or  requirements.  If  the  PM’s  program  has 
sufficient  funds  to  support  a  modification,  the  material  developer  creates  a  modification 
document  approved  by  the  combat  developer,  and  then  executed  by  the  equipment 
developer  under  the  guidance  of  the  PM.  If  the  PM’s  program  does  not  have  sufficient 
funds,  then  the  PM  has  a  choice  of  either  working  with  the  combat  developer  to  create 
possibly  a  new  requirement  or  to  do  nothing  at  all.  If  the  issue  does  not  fall  within  the 
initial  requirement  for  the  equipment,  then  a  new  requirement  must  be  created  and 
approved  by  the  Army  J3/5/7  and  combat  developer.  At  this  point,  the  CDP  is  reinitiated 
from  the  initial  phase.^^ 

B.  MANEUVER  CENTER  OF  EXCELLENCE:  THE  SOLDIER  SURVEY 

The  Test  and  Evaluation  Office  (TEO)  within  the  MCoE  at  Eort  Benning,  Georgia 
uses  a  comprehensive  survey  called  The  Soldier  Survey  to  gather  data  from  recently 
redeployed  soldiers  on  equipment  and  training.48  The  Soldier  Survey,  formally  known  as 
the  Post-Combat  Survey,  has  been  in  use  within  the  U.S.  Army  since  2003  with  18,000 
soldiers  participating  to  date.  The  MCoE  initiative,  which  is  “designed  to  assess  current 
field  support,  recommend  improvements,  and  inform  future  modernization  efforts  across 
DOTMEPE,”  allows  soldiers  to  provide  feedback  on  equipment  and  training  that  is  and  is 

46  Test  &  Evaluation  Office,  “The  Soldier  Survey  Information  Brochure”  (Fort  Benning,  GA: 
Maneuver  Center  of  Excellence,  2012),  http://www.benning.army.mil/Library/content/TheSoldier 
SurveyInformationBrochure.pdf. 

47  Bryan,  PEO  soldier  interview. 

48  Test  &  Evaluation  Office,  “The  Soldier  Survey  Information  Brochure.” 
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not  working .49  These  feedbacks  are  intended  to  assist  the  senior  U.S.  Army  leadership 
and  PMs  in  making  procurement  or  modification  decisions  within  the  CDP. 

The  Soldier  Survey  process  starts  with  requirements  as  outlined  by  MCoE,  PMs, 
U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  and  U.S.  Army  staff.  These 
requirements  focus  on  system  improvements,  materiel  development,  future 
modernization,  and  prioritization. 50  The  TEO  refines  the  requirements  into  questions  and 
administers  the  surveys  to  soldiers  using  laptops  in  a  classroom  setting  during  unit 
“lessons  learned  weeks.”  The  data  collected  is  then  analyzed  and  compiled  into  a  report 
called  The  Soldier  Survey  Report.  The  process  is  intended  to  be  cyclical  with  new 
requirements  being  derived  from  the  reports. 

The  format  of  The  Soldier  Survey  uses  Eikert  Scales,  multiple  choices,  and 
opened  ended  questions  to  ask  soldiers  to  rate  the  effectiveness  or  importance  of  U.S. 
Army  systems  (equipment  and  training). 5 1  The  survey  does  not  focus  on  a  single  or 
common  themed  system,  but  rather  covers  a  large  variety  of  systems  that  include 
mounted  systems,  clothing  items,  weapon  systems,  munitions,  optics,  and  sustainment 
items,  as  well  as  training  and  combat  operations. 52  The  output  of  The  Soldier  Survey 
provides  both  quantitative  and  qualitative  analysis  of  the  data  with  a  heavy  focus  on 
percentage  breakdowns  of  soldier  responses,  e.g.,  effectiveness  of  weapons  systems, 
comfort  of  clothing  items,  and  usefulness  of  particular  training.  The  output  also  includes 
bulleted  comments  from  the  open-ended  questions  that  highlight  the  quantitative  analysis. 

Through  a  multi-system  approach.  The  Soldier  Survey  has  been  able  to  cover  a 
large  field  of  requirements  in  a  short  amount  of  time  for  the  benefit  of  the  U.S.  Army 
senior  leadership  and  decision  makers.  The  Soldier  Survey  cycle,  methodology,  and 
content  has  resulted  in  significant  improvements  with  various  systems,  such  as  the  OCP 


49  Test  &  Evaluation  Office,  “The  Soldier  Survey  Information  Brochure.” 

50  Ibid. 

51  Ibid. 

52  Test  &  Evaluation  Office,  The  Soldier  Survey  Report  (Fort  Benning,  GA:  Maneuver  Center  of 
Excellence,  2013). 
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uniforms,  soldier  plate  carrier  system,  ground  combat  vehicle,  and  the  individual 

carbine.  53 

C.  WARFIGHTER  TECHNOLOGY  TRADESPACE  METHODOLOGY 

Within  the  DOD,  the  PM  executes  operational  assessments  through  the 
Operational  Test  Command  (OTC)  to  inform  the  CDP  process. 54  Operational  tests  are  not 
only  conducted  by  DOD  organizations,  they  are  also  conducted  by  independent 
organizations,  one  of  which  is  known  as  the  Adaptive  Red  Team  (ART).  ART  conducts 
operational  assessments  at  Technical  Support  and  Operational  Analysis  (TSOA),  which 
includes  both  government  and  industry  member  participations.  ART/TSOA  uncovers 
field  performance  and  usability,  as  well  as  system  vulnerabilities. 

ART/TSOA  works  with  and  informs  industry,  academia,  and  government 
laboratories  to  address  innovative  technologies  and  capabilities  for  the  warfighter.55 
ART/TSOA  identifies  candidate  technology  solutions,  informs  future  science  and 
technology  (S&T)  investments,  and  generates  insights  into  technology  robustness  and 
product  improvement  through  scenario-driven  exercises,  soldier  involvement,  and 
assessment  guided  by  the  WTTM. 

WTTM  is  the  academic  underpinning  of  TSOA.  Its  uses  practices,  tools,  and 
techniques  gained  from  systems  engineering,  logistics  and  human  systems  integration, 
and  modeling  and  simulation  to  support  quantitative  and  qualitative  assessments  to 
identify  new  system  vulnerabilities  and  risks.  WTTM  gains  feedback  on  equipment  by 
focusing  on  three  primary  domains:  logistics,  user,  and  technology.  WTTM  applies  three 
mechanisms  to  gain  feedback:  the  ART/TSOA  WTTM  Assessment  Tool,  the 
Hierarchical  Task  Analysis  (HTA),  and  informal  dialogues  between  evaluators,  service 


53  Test  &  Evaluation  Office,  “The  Soldier  Survey  Information  Brochure.” 

54  Operational  Test  Command  is  responsible  for  planning,  conducting,  and  reporting  operational  tests, 
assessments,  and  experiments  on  military  equipment.  OTC  is  a  subordinate  command  within  the  Army  Test 
and  Evaluation  Command  (ATEC).  U.S.  Army  Operational  Test  Command,  “United  States  Army 
Operational  Test  Command,”  February  28,  2014,  http://www.otc.army.mil/. 

55  Adaptive  Red  Team,  “2014-RFI  for  Adaptive  Red  Team/Technical  Support  Operational  Analysis 
(ART/TSOA)  Activity  Supporting  the  Deployable  Force  Protection  Technical  Focus  Team  (DFP  TFT) — 
2014-RFI-ART-TSOA  (Archived) — Federal  Business  Opportunities,”  FBO.gov,  October  13,  2013. 
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members,  and  equipment  developers.  The  ART/TSOA  WTTM  Assessment  Tool  (see 
Appendix  B,  Example  of  the  ART/TSOA  Assessment  Tool  (WTTM  Logistic  Factors)) 
uses  numerical  scales  to  evaluate  equipment  performance  (see  Figure  2).  The 
Hierarchical  Task  Analysis^^  Appendix  C,  Example  of  Hierarchical  Task 

Assessment  for  Centurion  Multi-Mission  System  [CMMS])  assesses  the  completion  time 
of  specified  tasks  by  either  expert,  trained,  or  untrained  individuals.  Informal  dialogues 
are  used  to  gain  evaluators,  service  members,  and  equipment  developers’  personal 
experiences  with  assessed  equipment.  Lastly,  all  three  feedbacks  mechanisms  collect 
commentaries  from  evaluators,  service  members,  and  equipment  developers. 


WTTM  User  Faclors 

Factor  1:  Training  Burden  (UB| 

Type  of  Trainer  Required 

Geographic  Location  of 
Required  Training 

Required  Time  to 
Train 

Notes  &  Comments 

EL 

ML 

EH 

None 

No  training  reouired 

0 

Some  systems  can  have  training  installed  in  parallel  with  its 
command  and  control  so  that  an  end  user  can  switch  to  this 
mode  as  needed.  The  chances  of  this  asset  getting  lost, 
damaged,  or  destroyed  is  less  than  if  it  is  shipped  with  the 
system. 

10 

10 

10 

Imbedded  on  System 

Location  doesn't  matter 

less  than  1  hour 

9 

9 

9 

Shipped  with  System  (e.g., 
DVD) 

1-2  hours 

8 

8 

8 

2-4  hours 

7 

7 

7 

Military  trainer  from 

within  unit 

is  installed  and 

operational 

4-6  hours 

6 

6 

6 

6-12  hours 

5 

5 

5 

Military  trainer  from 

outside  unit 

At  the  next  higher  unit's 

location 

12  -  24  hours 

4 

4 

4 

1-2  days 

3 

3 

3 

Field  Service 

Representative  Only 

CONUS  military  facilities 
only 

3-5  days 

2 

2 

2 

CONUS  developer  site(s) 
only 

more  than  a  week 

1 

1 

1 

Figure  2.  An  example  of  the  ART/TSOA  WTTM  assessment  tool 

assessing  the  domain  of  user  factors 


Through  TSOA  events,  WTTM  has  been  successful  in  improving  many  systems. 
Highlights  include  recommendations  that  reduced  the  size,  weight,  and  power  of  a  radar 
system,  reduced  system  vulnerabilities  through  encrypted  integration  into  situational 
awareness  software,  reduced  user  training  requirements  through  better  interface  design 
and  appearance,  as  well  as  the  addition  of  help  videos,  decreased  system  decision  time 
via  suggested  changes  to  user  interface,  increased  training  effectiveness  by  suggesting 
better  organization  and  design  of  soldier  training  devices,  etc. 


Hierarchical  Task  Analysis  (HTA)  is  a  process  of  developing  a  description  of  a  task  in  terms  of 
operations — things  which  people  do  to  attain  goals — and  plans — statement  of  conditions  when  each  of  a  set 
of  operations  has  to  be  carried  out  to  attain  an  operational  goal.  Barry  Kirwan  and  Les  K.  Ainsworth,  A 
Guide  to  Task  Analysis:  The  Task  Analysis  Working  Group  (Boca  Raton,  FL:  CRC  Press,  2003) 
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WTTM  parallels  the  CDP,  specifically,  the  operational  assessment  within  the 
production  and  deployment  phase  (see  Figure  3).  Currently,  in  phase  4,  the  PM  uses  the 
OTC  to  conduct  operational  assessment  of  new  equipment.^^  Similarly,  ART/TSOA 
provides  a  comprehensive  and  validated  option  to  the  equipment  developer  to  conduct 
operational  assessment  using  WTTM,  which  poses  the  question,  “Can  sensemaking  of 
service  member  narratives  as  a  feedback  mechanism  during  operational  assessments 
provide  useful  information  and  insights  for  the  equipment  developers  and  PMs?”  Given 
the  parallel  relationship  between  the  CDP  and  WTTM,  sensemaking  of  narratives  offers 
another  approach  for  assessing  equipment,  as  well  as  the  development  process  itself. 


Department  of  Defense 

i 


Figure  3.  Implementation  of  Operational  Assessment  of  Equipment  during 
Phase  4  of  the  CDP  and  during  the  ART/TSOA  of  WTTM 


U.S.  Army  Operational  Test  Command,  “United  States  Army  Operational  Test  Command.” 
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IV.  METHODOLOGY 


This  chapter  provides  the  method  by  which  this  research  is  conducted,  a  detailed 
description  of  the  collection  instrument  used  to  collect  service  member  narratives,  and  the 
computer  software  used  to  analyze  those  narratives. 

A.  PARTICIPANTS 

The  test  subjects  for  this  research  consists  of  U.S.  Army  (269  subjects)  and 
Marine  Corps  (10  subjects)  service  members  from  three  separate  battalions.  Two  U.S. 
Army  infantry  battalions  and  one  U.S.  Marine  Corps  battalion  were  selected  based  on 
their  availability  within  the  rest- training-deployment  cycle.  Service  members  were 
selected  based  on  purposive  random  sampling  of  each  battalion’s  demographic;  they  were 
included  in  the  sample  test  subjects  because  of  their  “specialist  knowledge  of  the  research 
issue,”  which  is  the  use  of  military  equipment. 58  The  data  collection  was  conducted  on 
three  separate  occasions  in  three  different  locations  from  May  to  December  2013. 

No  discriminating  factors  were  used  to  disqualify  the  test  subjects  from 
participating  in  the  research  (e.g.,  education  levels,  experiences  in  the  military,  or  MOS). 
As  long  as  the  test  subjects  were  part  of  the  units  sampled  and  were  willing  to  volunteer, 
they  were  free  to  participate.  Similarly,  the  test  subjects  were  informed  that  they  could 
discontinue  their  participation  at  any  time,  even  after  they  had  provided  their  input.  No 
service  members  discontinued  their  participation. 

To  protect  the  test  subjects’  privacy,  all  data  collected  from  the  service  members 
were  reviewed  for  any  personal  information  in  accordance  with  the  Naval  Postgraduate 
School  Institutional  Review  Board  (IRB)  protocol.  Additionally,  any  information 


58  Purposive  random  sampling  is  defined  as  a  form  of  non-probability  sampling  in  which  decisions 
concerning  the  individuals  to  be  included  in  the  sample  are  taken  by  the  researcher,  based  upon  a  variety  of 
criteria  that  may  include  specialist  knowledge  of  the  research  issue,  or  capacity  and  willingness  to 
participate  in  the  research.  Paul  Oliver  and  Victor  Jupp,  “Purposive  Sampling,”  2006,  http://eprints. 
hud .  ac  .uk/id/eprint/2049 . 
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identifying  specific  unit,  location,  and  mission  was  removed  in  accordance  with  the  DOD 
operational  security  (OPSEC)  policy  to  prevent  leakage  of  sensitive  information. 59 

B.  COLLECTION  AND  ANALYSIS  PROCESS 

We  administered  a  collection  instrument  using  a  PowerPoint  template  (see 
Appendix  D,  PowerPoint  Collection  Tool  Administered  To  Gather  Service  Member 
Narratives)  that  mirrored  the  SenseMaker^o  Collector,  which  was  comprised  of  a 
narrative  prompt.  Triads,  and  Dyads. We  then  transferred  the  PowerPoint  collected  data 
into  the  SenseMaker  Collector  using  12  computers  in  a  controlled  environment  to  capture 
narratives  and  their  signifiers.  We  provided  the  service  members  a  brief  introduction  to 
the  purpose  of  the  research  and  the  collection  instrument.  At  the  beginning  of  the 
collection  instrument,  service  members  received  a  narrative  prompt  (see  Figure  4)  that 
invoked  emotions  to  elicit  personal  experiences  with  equipment.  Service  members  were 
not  limited  to  answering  all  or  any  of  the  questions  presented  in  the  narrative  prompt. 
Service  members  then  created  their  narratives  by  making  sense  of  their  experiences  about 
equipment. 


59  Department  of  Defense,  DOD  Operations  Security  (OPSEC)  Program:  DOD  Directive  5205.02E, 
Department  of  Defense,  June  20,  2012. 

SenseMaker®  is  a  pattern  and  visual  analysis  tool  that  utilizes  both  hard  (numbers/facts)  and  soft 
(opinions/feelings)  data  for  examination.  David  Snowden  and  Steven  Dealing,  Computer-Aided  Methods 
and  Systems  for  Pattern-Based  Cognition  from  Fragmented  Material,  October  4,  2011. 

A  Triad  is  a  three-dimensional  signifier  designed  to  amplify  a  respondent’s  narrative;  a  Dyad  or  a 
linear  scale  signifier,  is  a  polarity  scale  that  amplifies  a  respondent’s  narrative.  “Introducing 
SenseMaker®,”  video  clip,  YouTube,  2009,  http://www. youtube. com/watch?v=SkRe7Xg7pk4& 
feature=youtube_gdata_player. 
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In  the  next  few  slides,  describe  an  event  with  the  newly  fielded  equipment  that  you  were  a 
part  of,  that  you  witnessed,  or  that  you  know  about.  Talk  about  one  day,  one  incident,  or  a 
personai  experience.  What  happened?  What  came  out  of  this  event?  What  would  you  tell 
others  who  would  be  using  that  equipment  in  the  future?  What  would  you  tell  the  equipment 
developers  that  best  conveys  your  satisfaction  or  dissatisfaction  with  the  new  equipment? 
What  do  they  need  to  know?  How  easy  or  difficult  was  it  to  adapt  to  the  new  equipment? 
What  were  the  particulariy  memorable  aspects  of  the  new  equipment? 

Please  tell  us  your  story  in  the  box  below 


Please  give  your  story  a  title  in  the  box  below: 


Figure  4.  Prompt  administered  to  service  members  to  collect  their 

narratives 


Upon  completion  of  the  narrative  portion,  service  members  completed  a  series  of 
nine  Triads  to  amplify  their  narratives.  Each  Triad  is  comprised  of  a  specific  prompt 
based  on  a  theme,  such  as  “improve”  (see  Appendix  E,  Eight  Triad  Themes,  Prompts,  and 
Signifiers),  and  three  signifiers.  Service  members  positioned  a  dot  within  the  Triad  that 
linked  the  relationship  between  the  three  signifiers  and  their  narrative.  If  no  relationship 
existed,  then  service  members  selected  the  “N/A”  option  (see  Eigure  5).  Triads  provided 
context  to  the  service  member  narratives  to  allow  a  deeper  understanding  of  their 
experiences.  Additionally,  Triads  allowed  the  service  members  the  ability  to  reinforce 
certain  ideas  and  experiences  within  their  narrative. ^2 


62  “Introducing  SenseMaker®.” 
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The  new  equipment  in  your  story  improved... 

Reduced  demands  on  your  time 


Click  and  drag  the  bubble  inside  the 
triangle  to  a  position  that  best  describes 
your  answer  relative  to  the  story  you  have 
just  told 

Select  N/A  if  the  triad  is  not  relevant  to  your 
experience 


Security 


Unit  morale 


ClN/A 


Figure  5.  An  example  of  a  SenseMaker  Collector  Triad  based  on  the  theme 
of  “Improve”  with  the  prompt  “The  new  equipment  in  your  story 

improved” 


In  a  manner  similar  to  Triads,  Dyads  provided  context  to  the  service  members’ 
narratives  to  allow  a  deeper  understanding  of  their  experiences.  Service  members 
completed  a  series  of  five  Dyads  to  amplify  their  narratives.  Each  Dyad  is  comprised  of  a 
specific  prompt  based  on  a  theme  and  two  signifiers  on  a  dyadic  scale  (see  Appendix  F, 
Ten  Dyad  Themes,  Prompts,  and  Signifiers).  Service  members  positioned  a  dot  along  the 
dyadic  scale  that  linked  the  relationship  between  the  two  signifiers  and  their  narrative.  If 
no  relationship  existed,  then  service  members  selected  the  “N/A”  option  (see  Figure  6). 
Dyads  allowed  the  service  members  the  ability  to  reinforce  certain  ideas  and  experiences 
within  their  narrative. ^3 


^3  Introducing  SenseMaker®.” 
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Click  and  drag  the  bubble  inside  the  scale  to  a  position  that  best  describes  your  answer 
relative  to  the  story  you  have  just  told. 


In  your  story,  the  new  equipment  impressed  you  most  favorably  by... 


The  value  added  to 
the  unit's  capabilities 


The  impact  it  had  on 
many  other  facets  in 
the  unit 


Bn/A 

In  your  story,  the  new  equipment  disappointed  you  most  by... 

Its  inability  to  deliver 
on  expectations 


The  impact  it  had  on  ( 
the  unit 


Bn/a 


Figure  6.  Examples  of  SenseMaker  Collector  Dyads  based  on  the  theme  of 
“Impressed”  and  “Disappointed”  with  the  prompts  “The  new 
equipment  impressed  you  most  favorably  by”  and  “The  new 
equipment  disappointed  you  most  by,”  respectively 


Upon  completion  of  the  collection  instrument,  the  aggregated  data  was  analyzed 
using  the  SenseMaker  Explorer  software  program,  which  provides  eight  functions  to 
analyze  the  data  on  two  levels,  pattern  and  narrative  analyses. xhe  cluster  and  distribute 
functions  (see  Eigure  7)  assist  in  identifying  areas  of  interest  or  concern  based  on  the 
disposition  of  service  member  narratives  depicted  as  dots  in  the  Triads  (see  Eigure  8)  and 
histogram  bars  in  the  Dyads  (see  Eigure  9).  We  then  extract  individual  or  groups  of 
narratives  from  the  areas  of  interest  and  read  them  to  gain  information  and  further 
insights  through  the  lens  of  different  themes.  This  software  program  allows  us  to  analyze 
the  data  that  “adds  layers  of  meaning”  rather  than  simply  read  the  narratives  at  the 
surface  level. 

SenseMaker®  Explorer’s  eight  functions  to  analyze  the  data  are  glance,  browse,  compare,  range, 
distribute,  cluster,  graph,  and  landscape.  “SenseMaker  Explorer;  Distribute  and  Cluster,”  video  clip, 
YouTube,  2010,  http;//vimeo. com/16278489. 

Cognitive  Edge  Pte  Ltd.,  “SenseMaker®,”  2013,  http://www.sensemaker-suite.com/smsite/ 
index,  gsp. 
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Figure  7.  SenseMaker  Explorer’s  eight  analysis  funetions 
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Figure  8.  The  Triad  is  an  example  of  SenseMaker  Explorer’s  eluster 
function  output  based  on  the  theme  of  “Improve.”  Each  dot 
represents  individual  service  member  narratives. 
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\  jt  P  a  (A 


Survivability 
-  So  good 
that  you  and 
your  unit 
could  always 
rely  on  it 


Survivability 

-So 

burdensome 
or  fragile  that 
no  one  can 
count  on  it 


Figure  9.  The  histogram  above  is  an  example  of  SenseMaker  Explorer’s 
distribute  function  output  based  on  the  theme  of  “Survivability.” 
Each  bar  within  the  histogram  represents  individual  narratives. 


SenseMaker  Suite’s  Collector  and  Explorer  provide  the  ability  to  collect  and 
analyze  quantitative  and  qualitative  data.  The  themes  used  during  the  research  could 
amplify  service  member  narratives  by  providing  context  and  a  deeper  understanding  of 
the  information  presented.  In  the  following  chapter,  the  data  is  analyzed  using 
Sensemaking  Explorer  to  determine  if  sensemaking  of  narratives  can  inform  the  Army 
CDP. 
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V.  RESULTS 


This  chapter  analyzes  a  dataset  comprised  of  279  service  member  narratives  and 
associated  responses  to  signifying  questions  using  the  SENSI  model  to  make  sense  of 
these  narratives  to  gain  information.  We  organized  the  narratives  into  two  groups  for 
analysis,  1)  the  collective  group  comprised  of  all  narratives,  and  2)  a  subset  focused  on 
the  GROSS. 66  We  analyze  the  narratives  on  two  levels.  The  first  level  uses  Triads  and 
Dyads  to  analyze  patterns  created  by  service  member  significations  (or  amplifications)  of 
their  narratives  in  response  to  a  specific  prompt  associated  with  a  theme.  Pattern  analysis 
allows  us  to  identify  specific  areas  of  interests  for  further  investigation.  The  second  level, 
narrative  analysis,  offers  additional  layers  of  meaning  and  understanding  of  service 
member  experiences  with  equipment.  Narrative  analysis  was  not  automatic.  We  did  not 
use  a  semantic  analysis  package.  Instead,  it  was  necessary  to  read  and  understand 
participants’  stories  in  addition  to  the  analysis  SenseMaker  might  provide.  Participants’ 
stories  spanned  a  broad- spectrum  of  operational  system  and  equipment  use.  As 
demonstrated  in  the  analysis  that  follows,  those  narratives  are  quite  informative 
regardless  of  sample  size  or  spectrum. 

A.  COLLECTIVE  GROUP  ANALYSIS 

This  section  analyzes  279  narratives  representing  90  different  pieces  of  equipment 
(see  Appendix  G,  Equipment  Eist  According  to  Primary  Eunctions  of  Shoot,  Move, 
Communicate,  and  Survive  According  to  Eield  Manual  7-21.13,  The  Soldier’s  Guide) 
using  the  Triad  theme  “Tailed  to  Improve”  and  Dyad  theme  “Implement  and  Operate.” 
Eigures  10  through  13  display  the  summary  results  of  service  member  input  and 
signification  in  response  to  narrative  interview  prompts. 


66  All  equipment  in  the  research  can  be  categorized  into  one  of  four  combat  tasks:  shoot,  move, 
communicate,  and  survive,  according  to  Field  Manual  7-21.13,  The  Soldier’s  Guide.  Headquarters 
Department  of  the  Army,  “FM  7-21.13  The  Soldier’s  Guide,”  Headquarters  Department  of  the  Army, 
February  2004,  http://armypubs.army.miFdoctrine/DR_pubs/dr_a/pdf/fm7_21xl 3.pdf 
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The  new  equipment  m  your  etoty  (ailed  to  impnm 


Reduced  demands  on  your 
time 


Figure  10.  This  Triad  is  based  on  the  theme  “Failed  to  Improve”  and  shows 
six  eoncentrated  areas  of  service  member  narratives.  Each  blue  dot 
corresponds  to  service  member  assessments  of  the  equipment 
described  in  the  narrative  as  it  relates  to  the  Triad  prompt  and  the 
meanings  associated  with  the  triangle’s  vertices. 

Figure  10  depicts  an  example  of  the  information  produced  by  SenseMaker 
Explorer,  by  capturing  service  member  signification  of  their  narratives  to  the  theme 
‘Tailed  to  Improve”  by  responding  to  the  prompt:  “The  new  equipment  in  their  story 
failed  to  improve...”  Data  displayed  in  Eigure  10  reflect  responses  from  279  service 
members  illustrating  how  the  equipment  that  they  described  in  their  narratives  fit  the 
prompt.  Due  to  the  diverse  spectrum  of  operational  equipment  and  a  large  volume  of 
service  members’  signification  of  their  narratives,  the  resulting  dataset  in  the  Triad  might 
be  expected  to  be  spread  equally  among  the  three  signifiers  (security,  unit  morale,  and 
reducing  the  demand  on  you  time)  on  the  vertices.  However,  our  analysis  of  Eigure  10 
suggests  that  the  significations  of  narratives  are  associated  more  with  security,  reduced 
demands  on  your  time,  or  both,  rather  than  unit  morale.  More  interestingly,  further 
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investigation  showed  that  the  heaviest  concentration  (31.2%)  of  representative  responses 
was  associated  with  the  signifier,  “security.”  This  piqued  our  interests,  which  required 
focusing  on  this  area.  Using  the  second  context  of  the  SENSI  model  to  analyze  narratives, 
we  gained  insight  into  why  warfighters  assessed  that  fielded  equipment  failed  to  improve 
security  in  Figure  1 1 . 


Figure  1 1 .  SenseMaker  Explorer  output  from  Figure  10,  but  now  with 

service  member  narratives  associated  only  with  signifier  “security.” 


Figure  11  depicts  the  identical  information  as  Figure  10,  but  now  with  selected 
service  member  narratives  identified  with  their  associated  concentration  and  the  signifier, 
“security.”  Participants’  commentaries  convey  that  equipment  failed  to  improve  security 
for  two  reasons,  1)  a  lack  of  training,  and  2)  its  inability  to  perform  claimed  capabilities. 
Service  members  said  that  equipment  failed  to  improve  security  because  of  a  lack  in 
training  on  the  system  and  the  inability  to  troubleshoot  the  system.  For  example,  several 
service  members  operating  the  Common  Remotely  Operated  Weapon  (CROW)  system 
stated  that  they  “had  trouble  working  with  the  Mark- 19  (40  mm  grenade  launcher)  on  the 
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system  because  of  link  rounds  not  feeding  properly.”  It  is  interesting  that  this  narrative 
identifies  at  least  two  possible  reasons  why  the  equipment  failed  to  improve  security. 
First,  the  service  member  could  have  failed  to  receive  proper  training  on  loading  and 
troubleshooting  the  weapon  system.  Second,  the  weapon  system  or  its  munitions  could 
have  experienced  a  technical  problem.  Analysis  of  additional  narratives  within  the 
concentration  in  Figure  11,  using  the  SENSI  model,  provides  more  insight  and  greater 
detail  to  why  equipment  failed  to  improve  security  than  simply  the  pattern  analysis. 

Representative  responses  displayed  in  Figure  11  stated  that  the  ACU  and  the 
THOR  III  Man  Portable  Counter  Radio  Controlled  Improvised  Explosive  Device 
Countermeasure  System  failed  to  perform  claimed  capabilities  of  the  fielded  equipment. 
Participants  stated  that  the  ACU  pattern  did  not  improve  their  security  because  it  failed  to 
adequately  provide  concealment  compared  to  the  multi-camouflage  pattern.  In 
comparison,  the  multi-camouflage  pattern  was  effective  in  “keeping  you  hidden  from  the 
enemy.”  The  ACU  pattern  did  not  increase  service  members’  survivability  or  improve 
their  security  level  against  enemy  combatants  in  a  desert  or  woodland  terrain. 

In  addition,  the  THOR  III  failed  to  improve  security,  which  resulted  in  service 
members  losing  confidence  in  the  system.  Eor  example,  subjects  stated  that  they  could 
“only  think  of  one  occasion  where  it  worked  and  many  times  when  it  failed.”  We  learned 
that  the  THOR  III  is  a  good  piece  of  equipment  in  concept  but  poorly  designed  because 
“it  overheats  often  and  can’t  be  carried  with  anything  else  on  your  back.”  The  narratives 
provide  insight  into  why  equipment  deficiencies  failed  to  improve  security. 


Eigure  12.  This  Dyad  is  based  on  the  theme  “Implement  and  Operate,”  and 
signification  of  associated  narratives  in  this  histogram  is  skewed  to 
the  right  showing  a  stronger  association  with  “difficult  for  anyone 
except  trained  specialists.” 


32 


Next,  in  Figure  12,  the  SenseMaker’s  distribute  function  is  used  to  analyze  a 
Dyad,  which  is  associated  to  the  theme  “Implement  and  Operate.”  Figure  12  illustrates 
the  signification  of  representative  responses  to  the  prompt  “The  skills  necessary  for  the 
user  to  correctly  implement  and  operate  the  new  equipment  in  your  story  were...”  The 
pattern  in  Figure  12  demonstrates  that  their  responses  were  skewed  significantly  toward 
the  difficult  (right)  end  of  the  response  spectrum.  Generally,  the  equipment  fielded  to  the 
participants  appeared  to  be  difficult  to  implement  and  operate. 


Figure  13.  Portions  of  narratives  associated  with  the  Dyad  in  Figure  12 
amplify  service  members’  significations  to  the  theme  “Implement 

and  Operate.” 


During  our  analysis  of  narratives  associated  with  the  histogram,  the  SENSI  model 
assisted  us  in  interpreting  why  the  equipment  was  difficult  to  implement  and  operate. 
Information  drawn  from  the  narratives  identifies  that  equipment  end-users  could 
implement  and  operate  the  equipment  but  only  at  the  rudimentary  level  because  of  lack  of 
training  associated  with  technical  aspects  of  the  equipment.  For  example,  service 
members  stated  that  the  Improved  Target  Acquisition  System  (ITAS)  was  capable  of 
operating  at  a  higher  proficiency,  but  they  lacked  the  required  training.  They  stated,  “I 
was  unable  to  effectively  use  it,  I  believe  with  additional  training,  I  would  be  able  to 
compensate  for  the  shortcoming.”  Furthermore,  additional  narratives  displayed  in  Figure 
13  state  that  service  members  used  trial  and  error  to  compensate  for  their  lack  of  training 
and  knowledge  of  the  ITAS.  Interestingly,  further  analysis  of  narratives  makes  it  possible 
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to  understand  that  the  ITAS’s  computer-based  operating  system  was  too  complex. 
Moreover,  the  narratives  specified  that  the  equipment  was  “very  technical  and  has  lots  of 
options  for  different  selections... the  system  was  not  very  user  friendly.”  Overall,  we 
learned  that  respondents  prefer  more  intuitive  equipment  to  implement  and  operate. 

Significations/amplification  of  narratives  in  Figures  10  and  12  combined  with 
associated  narrative  fragments  in  Figures  11  and  13  provide  valuable  insights  that  can 
inform  the  equipment  development  and  fielding  process.  Our  analysis  of  narratives  in 
Figures  10  and  11  conveyed  that  service  members  were  primarily  concerned  about  their 
security  and  the  acceptance  of  the  equipment  based  on  its  performance.  Figures  12  and  13 
show  that  an  ill-conceived  computer-based  operating  system  and  a  lack  of  training  were 
not  optimal  for  warfighters  to  implement  and  operate.  Narratives  associated  with 
concentrated  patterns  in  both  Figures  11  and  13  indicated  that  if  the  obstacles  associated 
with  training,  performance,  and  technical  aspects  of  the  equipment  could  be  overcome 
prior  to  fielding,  then  they  could  be  better  received  by  service  members. 

Given  a  diverse  collection  of  equipment  and  a  large  volume  of  service  members’ 
signification  of  their  narratives.  Army  General  Staff,  specifically  the  G3/5/7,  could  use 
the  information  derived  from  service  member  narratives  to  define  and  validate  equipment 
requirements  in  the  form  of  DOTMLPF.  This  information  could  assist  in  overcoming 
previously  identified  obstacles  from  Figures  11  and  12.  Also,  the  Army  G3/5/7  could 
provide  guidance  and  directives  to  address  issues  within  the  CDP  to  improve  future 
development  of  equipment.  The  information  derived  from  a  diverse  collection  of 
equipment  offers  insight  to  PMs  and  equipment  developers  but  is  limited  because  the 
information  is  generalized.  However,  using  the  SENSI  model  to  analyze  narratives 
associated  with  a  single  piece  of  equipment  might  provide  more  detailed  information  that 
could  inform  PMs  and  equipment  developers  for  potential  future  success  of  equipment 
development  and  fielding.  We  will  apply  this  process  to  make  sense  of  service  member 
narratives  associated  with  the  GBOSS/Cerberus/CMMS. 
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B.  GBOSS/CERBERUS  ANALYSIS 


This  section  analyzes  1 1  narratives  representing  a  specific  piece  of  equipment,  the 
GROSS — a  fielded  system  from  a  family  of  equipment  that  has  been  assessed  at  every 
TSOAAVTTM  event.  Although  we  were  only  able  to  analyze  a  small  sample  size  of 
participants  who  had  worked  with  the  GROSS,  the  results  proved  insightful.  Our  initial 
findings  applying  the  SENSI  model  indicate  that  narratives  provide  feedback  that  is  very 
similar  to  commentaries  from  the  ART/TSOA  WTTM  Assessment  and  HTA  feedback. 
We  gained  this  insight  by  analyzing  narratives  associated  with  the  GROSS  using  the 
Triad  themes  “Renefit  Provided”  and  “Aspects  to  Improve.”  We  expanded  our  research 
by  examining  stories  linked  to  the  Dyad  themes  “Maintaining  Equipment,”  and  “New 
Equipment  Integration.”  This  information  could  help  PMs  and  equipment  developers 
better  understand  the  GROSS  and  its  shortcomings.  In  addition,  this  information  could 
lend  insight  into  the  WTTM  process. 

The  GROSS,  also  known  as  Cerberus  or  CMMS,  is  a  force  protection  system  that 
provides  the  ability  to  detect,  track,  display,  record,  and  transmit  video.  Specifically, 
the  GROSS  offers  two  sensors,  a  motion  detection  radar  and  a  day/night  camera  system. 
The  radar  provides  a  360-degree  field  of  view  with  a  range  of  two  kilometers.  The 
camera  system  has  a  thermal  capability  for  limited  visibility  and  night  use,  as  well  as  a 
day  vision  camera.  The  system  provides  persistent  surveillance  using  radar  to  enhance  the 
ability  to  detect  moving  targets. The  GROSS  can  be  operated  in  two  configurations, 
trailer-mounted  as  shown  in  Eigure  14  and  man -portable  pack  shown  in  Eigure  15.  In  the 
following  narratives,  the  participants’  experiences  with  the  GROSS  are  examined  to  gain 
insight  into  the  development  and  fielding  of  the  system. 


William  D.  Midgette,  “Enhancing  the  Operational  Effectiveness  of  the  Ground-Based  Operational 
Surveillance  System  (G-BOSS)”  (master’s  thesis,  Naval  Postgraduate  School,  2008),  http://oai.dtic.mil/oai/ 
oai?verb=getRecord(&metadataPrefix=html&identifier=ADA483482. 

U.S.  Marine  Corps,  “Ground  Based  Operational  Surveillance  System,”  Marines,  accessed  April  17, 
2014,  http://www.pendleton.marines.mil/StaffAgencies/AssistantChiefofStaffG35/TrainingSupport 
Division/TrainingDevices/GroundBasedOperationalSurveillanceSystem.aspx. 
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Figure  14.  Trailer- mounted  GROSS  system  configured  for  operation 


U.S.  Marine  Corps,  “Ground  Based  Operational  Surveillance  System.” 
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Figure  15.  The  system  shown  is  a  GROSS  Lite  in  the  man-portable 

eonfiguration^o 


MTEQ,  Inc.,  “MTEQ — Manufacturing  Services  Cerberus  Lite,”  MTEQ,  2012,  http://www.mteq. 
com/cgi-bin/dynamic.py?Page=manufacturingServicesCerberusLite. 
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Figure  16.  Signification  of  narratives  to  the  prompt  “The  benefits  to  your 
unit  provided  by  the  new  equipment  was  in. . Blue  dots  correspond 
with  service  member  assessment  of  the  GBOSS. 


Figure  16  shows  the  resulting  signification  of  narratives  associated  with  the  theme 
“Benefit  Provided”  that  prompts  service  members  to  answer  what  “benefit  the  new 
equipment  provided  to  their  unit.”  The  pattern  represented  in  Figure  16  illustrates  that  all 
1 1  respondents  associated  their  narratives  with  the  signifier,”  safety  and  security,”  even 
though  two  of  these  respondents  also  associated  their  narratives  with  the  other  vertices. 
The  pattern  prompted  us  to  read  and  interpret  all  11  stories  to  gain  insight  into  why 
service  members  signified  that  the  GBOSS  benefited  safety  and  security.  Information 
gained  from  these  11  narratives  shows  that  the  GBOSS ’s  technical  capabilities  increased 
the  participants’  safety  and  security.  This  finding  is  consistent  with  ART/TSOA  WTTM 
Assessment  and  HTA  assessments  and  commentaries,  which  is  elaborated  upon  further  in 
this  section.  However,  the  narratives  might  offer  additional  insights  with  equipment 
capabilities  and  performance,  training,  and  fielding,  which  are  explored  further. 
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Figure  17.  Participant  signification  and  representative  responses  to  the 
signifying  prompts  “The  benefit  to  your  unit  provided  by  the  new 
equipment  was  in. . .” 


Figure  17  depicts  representative  responses  that  articulate  how  the  GROSS 
benefited  service  members’  safety  and  security  because  of  its  technical  capabilities.  The 
GROSS  provided  greater  situational  awareness  around  the  operational  environment 
through  its  enhanced  visual  monitoring  capability.  Participants  stated  that  they  could 
“look  at  situations  that  were  happening  in  the  surrounding  area... see  into  the  blind  spots 
of  the  post... see  in  places  that  some  of  the  posts  couldn’t.”  Additionally,  service 
members  stated  that  “the  capabilities  of  the  day  and  night  cameras  were  exceptional” 
providing  observation  up  to  “three  to  four  kilometers  away.”  This  information  is 
comparable  to  ART/TSOA  findings  that  confirm  the  warfighters’  use  of  the  GROSS’S 
visual  monitoring  capabilities  but  does  not  address  the  actual  performance.  For  example, 
one  ART/TSOA  evaluator  stated,  “User  is  able  to  analyze,  classify,  and  use  all  output 
products.”  The  signification  of  narratives  showed  that  the  GROSS,  possibly  its  visual 
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monitoring  capabilities,  benefited  their  safety  and  security.  More  importantly,  the  first 
context  of  the  SENSI  model  shows  how  narratives  capture  participants’  experiences  with 
the  GROSS  to  articulate  how  it  benefited  their  safety  and  security  along  with 
shortcomings. 

While  providing  feedback  on  the  capabilities  and  performance  of  the  GROSS, 
narratives  highlighted  possible  shortcomings  of  the  system.  End  users  stated  that  the 
GROSS  provided  the  ability  to  observe  enemy  targets  and  positions,  but  it  failed  to 
provide  enemy  grid  locations  accurately.  Eor  example,  they  stated,  “The  system  in  its 
current  configuration  will  allow  for  general  situational  awareness,  however  is  not  capable 
of  providing  any  precision  target  referencing  or  targeting  in  general... when  actually 
operating  the  system,  I  noticed  its  lack  of  ability  to  give  precise  grid  locations  of  potential 
targets  I  was  observing.”  This  feedback  aligns  with  commentaries  from  the  HTA, 
specifically  “slew  to  cue... obtaining  grid  locations  to  targets  was  not  obvious.”  However, 
these  capabilities  do  exist  within  the  GROSS.  The  disparity  between  the  feedbacks  and 
the  actual  capabilities  of  the  GROSS  raises  an  interesting  question:  “why  did  service 
members  not  know  the  complete  capabilities  of  the  GROSS?”  Eurther  investigation  into 
Eigures  18  and  19  may  possibly  provide  insights  into  whether  the  cause  of  the  limitation 
was  the  equipment  interface  or  a  training  issue  or  both  (most  likely). 
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Based  on  your  story,  what  aspect  of  the  ne  w  equipment  could  l>e  improved  the  most’ 


Soldier  training 

requirements 

Figure  18.  Signification  of  narratives  to  the  prompt  “What  aspect  of  the  new 

equipment  could  be  improved  the  most. . .” 


Figure  18  displays  a  Triad  signification  of  service  member  narratives  illustrating 

their  association  to  the  theme  “Aspects  to  Improve”  and  prompts  service  members  to 

answer  “what  aspect  of  the  new  equipment  could  be  improved  the  most?”  The  resulting 

signification  pattern  shows  that  service  member  training  requirements  could  be  improved 

the  most  while  the  left  and  right  ends  are  weighted  equally.  This  pattern  is  consistent  with 

our  findings  from  Figure  17  identifying  training  as  a  possible  issue  with  the  GROSS.  The 

consistency  in  the  two  findings  and  WTTM  feedback  possibly  offers  an  area  of  interest 

for  further  consideration  to  understand  how  soldier-training  requirements  could  be 

improved  the  most.  In  addition,  we  found  it  intriguing  that  the  left  and  right  vertices  were 

weighted  equally,  which  prompted  further  analysis.  Once  again,  information  gained  from 

stories  and  ART/TSOA  WTTM  Assessment  and  HTA  commentaries  remain  similar.  For 

example  ART/TSOA  evaluator  commentary  states  “there  is  (sic)  certainly  more  than  5 

steps  to  access  primary  functionality  of  the  system  and  requires  insights  that  are  beyond 
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the  average  Soldier,”  which  is  consistent  with  our  analysis  of  narratives  associated  with 
Figure  19. 


Figure  19.  Participant  signification  and  representative  responses  to  the 
signifying  prompt  “What  aspect  of  the  new  equipment  can  be 
improved  the  most?”  Participants’  narratives  convey  that  training 
requirements  is  the  most  important  aspect  to  improve. 


Information  gained  from  Figure  19  identifies  that  the  participants  did  not  receive 
any  formal  training  on  the  GROSS,  but  rather,  they  gained  experience  through  repetitive 
equipment  use  during  their  deployment.  A  subject  stated,  “I  received  literally  zero 
training  on  it  before  using  it  and  never  got  any  training  other  than  my  own  findings  over 
the  course  of  using  it.”  Additionally,  another  subject  noted,  “I  never  received  any  training 
on  the  GROSS  so  it  was  a  just  play  with  it  and  see  what  does  what.”  Although  they  were 
able  to  employ  the  equipment,  the  lack  of  training  prevented  them  from  operating  the 
GROSS  to  its  full  potential.  This  deficiency  highlights  a  possible  problem  with  the  Relief 
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in  Place/Transfer  of  Authority  (RIP/TOA)  process,  in  which  service  members  may  not 
receive  training  on  equipment  transferred  between  units.  One  interesting  fact  of  the 
WTTM  HTA  process  is  that  it  has  untrained  soldiers  perform  basic  functions  and 
operations  on  the  GROSS  during  ART/TSOA  events.  This  situation  is  a  parallel  endeavor 
to  the  RIP/TOA  process;  in  other  words,  soldiers  in  the  field  receive  no  handover  time 
with  equipment  that  they  inherit  after  relieving  a  unit.  More  importantly,  service 
members  stated  that  “there  were  a  lot  of  capabilities  it  (GROSS)  had  that  (they)  never 
understood,”  as  well  as  that  they  “didn’t  even  understand  what  half  the  buttons  or 
switches  did.”  These  narratives  provided  insights  that  users  employed  the  GROSS  to  less 
than  its  full  potential  because  they  lacked  formal  equipment  training  or  because  the 
GROSS  user  interface  was  so  burdensome  as  to  preclude  discovery  of  the  system’s  full 
potential. 

The  ART/TSOA  evaluator  commentaries  indicated  that  formal  training  is  required 
to  use  the  GROSS  to  its  full  potential,  similar  to  our  findings.  Moreover,  HTA 
commentaries  specified,  “the  system  assembly  was  not  an  issue  for  trained  service 
members  but  proved  difficult  for  those  untrained  due  to  a  high  number  of  components.” 
Also,  ART/TSOA  WTTM  Assessment  feedback  stated  that  “equipment  performance 
hinged  on  service  member’s  operational  training  on  the  GROSS  because  the  system 
interface  was  not  intuitive  and  required  too  many  steps.”  Interestingly,  the  comment 
referencing  system  interface  is  also  found  in  our  narrative  analysis. 

Interpretation  of  our  data  revealed  that  the  GROSS  interface  was  an  equipment 
shortcoming  for  warfighters  because  “the  system  was  not  very  user  friendly;  with  each 
additional  menu  that  was  opened,  that  window  would  dock  on  within  the  viewing  panel 
and  reduce  the  usable  screen  space.”  Comparable  to  the  narratives,  HTA  commentaries 
conveyed,  “the  GROSS’S  operational  screen  was  not  intuitive  to  service  members  when 
operating  the  system,”  and  “the  interface  was  not  similar  to  any  currently  fielded 
surveillance  systems.  One  of  the  soldiers  in  the  WTTM  HTA  field  exercise  and 
assessment  suggested  that  it  might  be  beneficial  to  have  the  interface  share  common 
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features,  appearances,  functionalities  with  Blue  Force  Tracker’s.”^!  Should  the  PM  and 
equipment  developer  have  an  interest  in  this  information,  these  narratives  and 
commentaries  could  inform  them  that  the  interface  is  too  difficult  to  operate  without 
proper  training,  which  could  negatively  affect  combat  performance.  This  explanation 
suggests  that  a  more  intuitive  interface  has  the  potential  to  increase  service  members’ 
ability  to  understand  and  operate  the  equipment;  thereby,  possibly  increasing  combat 
performance  even  with  a  lack  of  formal  training. 

While  the  results  from  this  analysis  of  the  top  vertices  from  Figures  18  and  19 
provide  valuable  information,  it  may  be  possible  to  gain  additional  information  by 
analyzing  the  narratives  associated  with  the  left  and  right  vertices  of  these  Triads. 
Examination  of  narratives  associated  to  the  signifier,  “equipment  combat  performance” 
and  WTTM  assessments  and  commentaries  showed  that  the  GBOSS  performed  sub- 
optimally  under  unfavorable  conditions.  Our  analysis  of  narratives  suggests  that  the 
GBOSS  had  problems  “when  weather  was  an  issue”  and  “continuously  froze  or  was 
blurry.”  In  the  case  of  ART/TSOA  WTTM  Assessment,  commentary  addresses  that  the 
GBOSS  underperformed  in  certain  unfavorable  external  conditions  (e.g.,  excessive 
vibrations  and  shocks).  It  was  learned  through  both  analyses  that,  1)  equipment 
assessment  might  benefit  from  testing  in  adverse  weather  conditions,  and  2)  PMs  and 
equipment  developers  may  consider  improving  the  design  of  the  GBOSS  to  withstand 
unfavorable  external  conditions. 

Interpretation  of  both  the  narratives  associated  to  the  signifier,  “logistic 
requirement”  and  WTTM  assessments  and  commentaries  showed  that  the  GBOSS 
maintenance  requirement  placed  added  burden  on  service  members  and  their  units  which 
led  to  the  question  of  what  kind  of  burden  the  participants  experienced.  Service  member 
narrative  commentaries  suggest  that  when  the  GBOSS  malfunctioned  or  became 
inoperable,  they  “had  to  wait  for  a  specialist  to  fix  it.”  Along  the  same  line,  WTTM 
assessment  feedback  stated  the  GBOSS  “uses  radars  and  camera  systems  that,  if  broken. 

Blue  Force  Tracking  (BFT)  provides  situational  awareness  capabilities  in  products  and  systems  that 
identify  and  track  “friendly  forces.”  General  Dynamics  C4  Systems,  “Blue  Force  Tracking,”  General 
Dynamics  C4  Systems,  accessed  April  4,  2014,  http://www.gdc4s.com/blue-force-tracking.html7taxonomy 
Cat=132. 
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could  cause  significant  delay  in  getting  the  system  back  operational.  This  could  take 
weeks  unless  the  technology  is  shipped  with  additional  sensory  equipment.”  Also,  “over 
the  course  of  the  TSOA,  the  CMMS  (GBOSS)  had  several  software  malfunctions  that 
required  a  SME  (Subject  Matter  Expert)  to  remedy.”  It  might  be  concluded  that  a 
participant’s  inability  to  fix  the  GBOSS  and  the  time  required  to  repair  the  system  limited 
its  use  and  reliability,  which  placed  a  significant  burden  on  the  unit.  However, 
commentaries  also  suggested  that  issues  occurred  within  the  development  and  fielding 
processes,  such  as,  1)  lack  of  training  for  end  users  on  maintaining  the  GBOSS,  2)  limited 
number  of  trained  specialists  to  support  the  system,  or  3)  the  units  may  possibly  lack 
adequate  tools  to  repair  the  deficiencies.  Surprisingly,  the  current  HTA  (see  Appendix  C, 
Example  of  Hierarchical  Task  Assessment  for  Centurion  Multi-Mission  System 
(CMMS))  employed  at  ART/TSOA  events  does  not  assess  the  end  user’s  ability  to 
troubleshoot  and  repair  system  deficiencies.  We  suggest  that  WTTM  incorporate 
troubleshooting  and  repair  as  tasks  when  assessing  soldiers  working  with  new  equipment 
at  TSOA  events. 

Overall,  the  insights  gained  from  the  Triad  in  Eigures  16-19  and  their  associated 
narratives  could  prove  useful  to  the  development  and  fielding  processes  by  identifying 
potential  capability  and  technological  gaps  that  might  have  been  overlooked,  as  well  as 
providing  solutions  to  fill  these  gaps.  We  now  turn  to  an  investigation  of  the  Dyads 
(Eigures  20-23)  associated  with  the  GBOSS  to  gain  additional  insights  into  improving 
the  overall  development  and  fielding  processes. 


The  skills  necessary  to  maintain  the  new  equipment  in  your  story  were  . 

Simple- 
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Eigure  20.  Dyad  results  from  participants’  significations  of  narratives 
regarding  “The  skills  necessary  to  maintain  the  new  equipment. . .” 
Histogram  depicts  that  special  training  was  needed  to  maintain  the 

GBOSS. 
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Figure  20  summarizes  service  members’  significations  of  their  narratives  in 
response  to  the  prompt:  “the  skills  necessary  to  maintain  the  new  equipment  were...” 
associated  with  the  theme  “Maintaining  Equipment.”  The  pattern  in  this  histogram  clearly 
illustrates  that  the  GROSS  is  difficult  to  maintain  because  it  required  special  training. 
Furthermore,  this  data  appears  to  be  consistent  with  and  possibly  validates  both  our 
results  from  Figure  19  and  WTTM  assessment  feedback  gained  through  TSOA  events. 


Figure  21.  Portions  of  narratives  associated  with  the  Dyad  in  Figure  20 
amplify  participants’  significations  to  the  theme  “Maintaining 

Equipment.” 


Figure  21  conveys  portions  of  narratives  that  support  our  previous  findings  in 
Figure  19,  specifically,  the  narratives  associated  with  the  signifier,  “logistic 
requirements.”  Our  analysis  of  narrative  segments  associated  with  the  Dyad  in  Figure  21 
states,  “when  it  (GROSS)  went  down  it  took  a  while  for  the  system  to  come  back  up  and 
if  it  was  serious  it  would  take  a  specialist  to  fix  it,”  and  the  “downfalls  of  the  GROSS  was 
the  fact  that  it  was  hard  to  maintain  and  keep  operating  in  the  cold.”  These  narrative 
fragments  reiterate  that  service  members  experienced  shortcomings  with  the  GROSS  that 
they  were  unable  to  correct  without  a  trained  specialist.  These  fragments  also  confirm  the 
WTTM  assessment  commentary,  which  stressed  the  same  logistics  concern.  Although  our 
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analysis  of  the  Dyad  did  not  provide  additional  insights  into  maintaining  the  GBOSS,  it 
did  reinforce  previous  findings  that  could  improve  the  system. 

Last,  we  analyze  the  manner  in  which  the  GBOSS  might  integrate  with  other 
combat  systems  and  the  operational  environment.  This  analysis  produces  useful 
information  that  could  enhance  the  CDP  and  WTTM  assessment  process.  Figure  22 
depicts  subjects’  signification  of  their  stories  associated  to  the  theme  “New  Equipment 
Integration”  and  prompts  a  response  from  them  regarding  “how  easy  was  it  for  you  and 
your  unit  to  integrate  the  GBOSS  with  other  combat  systems  and  the  operational 
environment?”  As  illustrated  in  Figure  22,  the  signification  demonstrates  that  service 
members  had  a  mixed  assessment  of  the  GBOSS ’s  integration  with  other  combat  systems 
and  the  operational  environment.  However,  further  examination  of  their  responses 
displayed  in  Figure  23  suggests  that  the  GBOSS  actually  integrated  easily  with  the 
operational  environment  and  other  combat  systems,  which  added  value  to  their  unit.  Our 
conclusion  mostly  aligns  with  observations  from  WTTM  assessments,  particularly, 
GBOSS  integration  with  other  combat  systems. 


How  easy  was  itforyou  and  your  unitto  integrate  the  new  equipment  with  other  combat  systems  and  the  operationai  environment? 


Figure  22.  Dyad  results  from  subjects’  significations  of  narratives  regarding 
GBOSS  integration  with  other  combat  systems  and  the  operational 

environment. 
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In  Afghanistan,  the  G-BOSS  system  was  used  to 
provide  an  extended  reach  of  observation,  from 
multiple  posts,  and  from  the  central  Operations 
Center.  As  well  as  providing  the  extended  view, 
it  enabled  us  to  continuously  search,  24  hours  a 
day,  7  days  a  week,  for  anything  suspicious, 
demeaning,  or  harmful.  During  our  operations 
with  the  G-BOSS  (and  V-BOSS)  systems  we 
were  able  to  track,  and  mark  certain  nefarious 
areas,  which  prevented  lED  strikes. 


The  GBOSS  was  helpful  for  seeing  in  places 
that  some  of  the  posts  couldn't  Whenever 
any  type  of  pop  shots  or  explosions  were 
heard  in  the  distance  it  was  good  to  get 
eyes  on  some  things  such  as  smoke  clouds. 
It  helped  the  civilians  on  the  blimp  to 
coordinate  with  us  when  we'd  find  a  smoke 
cloud  and  they  could  get  eyes  on  it  when  we 
couldn't 


Figure  23.  Portions  of  narratives  assoeiated  with  the  Dyad  in  Figure  22 
amplify  subjeets’  significations  to  the  theme  “Equipment 
Integration.” 


Service  members  expressed  that  they  were  able  to  integrate  the  equipment  quickly 
into  the  operational  environment  to  enhance  their  situational  awareness.  For  example, 
they  stated  that  the  GBOSS  “provided  an  extended  reach  of  observation,”  which  allowed 
the  service  members  to  “continuously  scan  the  perimeter  of  the  FOB  (Forward  Operating 
Base)  from  the  walls  out  to  the  nearest  house.”  Also,  the  stories  articulated  that  the 
GBOSS  provided  the  ability  “to  track  and  mark  certain  nefarious  areas,  which  prevented 
LED  (Improvised  Explosive  Devices)  strikes.”  The  narratives  indicated  that  the  increased 
safety  and  security  added  from  equipment  integration  within  their  operational 
environment  was  easy  to  notice. 

Additionally,  narratives  showed  that  units  were  integrating  the  GBOSS  with  other 
base  security  and  observation  equipment  in  complementary  roles.  A  participant’s  story 
identified  that  the  GBOSS  easily  integrated  with  another  force  protection  and  observation 
system  (an  observation  blimp)  in  Afghanistan.  A  warfighter  stated  that  the  GBOSS 
“helped  the  civilians  on  the  blimp  to  coordinate  with  us  when  we’d  find  a  smoke  cloud 
and  they  could  get  eyes  on  it  when  we  couldn’t.” 
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Similar  representative  responses  mirrored  some  of  the  results  from  WTTM 
assessments  and  commentaries  concerning  equipment  integration.  For  instance,  a  TSOA 
evaluator’s  commentary  identified  that  the  GROSS  “would  be  integrated  in  the  unit’s 
Operation  Center  by  the  FSR  (Field  Service  Representative)  upon  initial  setup.”  As  well, 
feedback  from  the  ART/TSOA  WTTM  Assessment  discussed  that  once  the  integration 
between  the  systems  was  established,  the  GROSS  performed  without  issues. 

Comparison  of  narratives  and  WTTM  commentaries  also  shows  that  narratives 
may  fail  to  capture  important  information  about  the  GROSS,  specifically  addressing  its 
assembly.  Our  analysis  of  collected  narratives  did  not  address  system  assembly  of  the 
GROSS  because  the  research  subjects  fell  in  on  already  assembled  systems.  However,  the 
WTTM  HTA  did  gain  feedback  on  the  GROSS  assembly,  which  annotated  the 
complexity  of  the  task  for  untrained  users.  Specifically,  the  HTA  commentary  described 
how  end  users  were  confused  with  the  layout  of  the  cables  and  were  unable  to  reference 
any  “Quick”  assembly  diagrams.  Further  commentary  from  the  ART/TSOA  WTTM 
Assessment  explained,  “once  trained  on  the  system,  soldiers  will  have  no  difficulties 
setting  up  the  system.”  We  learned  that  WTTM  feedback  mechanisms  are  capturing 
insightful  information  that  can  assist  in  the  future  development  and  fielding  of  the 
GROSS.  However,  we  also  conclude  that  sensemaking  of  narratives,  as  described  in  the 
second  context  of  the  SENSI  model,  either  as  a  stand-alone  process  or  a  complementary 
approach  with  existing  mechanisms,  can  inform  the  CDP  and  assist  WTTM  in  assessing 
equipment. 
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VI.  CONCLUSION 


A.  OVERALL  FINDINGS 

We  determined  that  sensemaking  of  service  member  narratives  associated  with 
fielded  systems  and  equipment  provided  valuable  insights  into  performance  expectations 
that  can  inform  both  the  CDP  and  WTTM.  We  obtained  stories  from  279  participants 
through  interviews  in  accordance  with  IRB  protocol  to  create  a  database  of  narratives  and 
signification.  To  analyze  this  data,  we  created  the  SENSI  model  that  made  it  possible  to 
study  soldier  narratives  that  they  created  from  their  experiences.  By  using  this  model,  we 
analyzed  soldiers’  signified  narratives  through  patterns,  narratives,  and  narrative 
fragments  to  draw  inferences  regarding  new  equipment  and  the  CDP.  The  SENSI  model 
provided  a  novel  approach  to  gain  information  about  WTTM  attributes  and  shortcomings 
(e.g.,  GBOSS  performance  in  cold  weather  and  having  soldiers  perform  maintenance  and 
troubleshooting  tasks  during  TSOA  events). 

Applying  this  innovative  process  to  gain  insights  about  fielded  equipment,  we 
were  able  to  identify  benefits  and  shortcomings  of  narratives  and  sensemaking  for  such 
purposes.  One  benefit  that  narratives  provide  is  firsthand  accounts  of  end  user 
experiences  rather  than  second  hand  interpretations  from  a  third  party  (e.g.,  evaluators, 
observers,  etc.),  which  could  contain  biases.  Moreover,  the  use  of  open-ended  prompts  to 
elicit  these  firsthand  accounts  can  further  decrease  unintended  biases  that  can  result  from 
a  structured  question  format  in  traditional  surveys.  Eurthermore,  we  found  that  open- 
ended  prompts.  Dyads,  and  Triads  offered  provocation  that  elicited  emotive  responses 
from  end  users  about  their  experiences  with  equipment. 

A  benefit  of  sensemaking  of  narrative  is  that  it  offers  a  more  in-depth  approach 
for  analyzing  participants’  experiences.  When  narratives  are  associated  with  a  theme  and 
signifiers,  they  provide  a  deeper  and  richer  understanding  than  that  accessible  through 
traditional  elicitation  approaches,  such  as  Eikert-based  surveys.  Through  sensemaking, 
researchers  are  able  to  gain  insights  and  additional  layers  of  meaning,  which  may  have 
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been  overlooked,  discarded,  or  unidentified.  This  information  could  be  used  to  overcome 
shortcomings  of  fielded  equipment. 

Although  narrative  analysis  provides  a  beneficial  approach  to  assessing 
equipment,  two  shortcomings  may  limit  its  potential  application  in  the  CDP  and  WTTM 
if  not  addressed.  First,  analyzing  narratives  is  labor  intensive  because  it  requires  human 
judgment  to  determine  if  the  information  presented  in  the  narrative  is  relevant  and  has 
value.  Although  an  automated  approach  to  simulate  human  judgment  does  exist  in  the 
theory  of  Latent  Semantic  Analysis  (LSA),^2  application  is  limited.  Researchers  in  the 
field  of  semantics  state,  “it  is  still  impossible  to  perform  SVD  (singular  value 
decomposition)  on  the  hundreds-of-thousands-by-tens-of-millions  matrices  that  would  be 
needed  to  truly  represent  the  sum  of  an  adult’s  language  exposure. Simply,  the  ability 
to  replicate  human  judgment  and  experiences  required  to  understand  narratives  fully 
through  automated  means  is  inadequate. 

Second,  analyzing  narratives  is  subjective  and  can  result  in  conflicting 
interpretations  based  on  researcher  perspectives.  Although  it  is  nearly  impossible  to 
remove  subjectivity  associated  with  narrative  analysis  completely,  it  is  possible  to 
minimize  it  through,  1)  an  increased  sample  size  of  narratives,  2)  multiple  researchers 
must  analyze  narratives,  or  3)  collect  and  analyze  supporting  quantitative  (statistical)  data 
associated  with  equipment  (e.g.,  completion  time  of  specific  task,  success/failure  rate  of 
performance,  etc.).  Researchers  should  apply  these  recommendations  to  minimize 
incongruent  interpretations.  If  these  shortcoming  are  not  properly  addressed,  PMs  and 
equipment  developers  risk  not  capturing  or  analyzing  the  rich  data  within  the  narratives 
to  inform  the  development  and  fielding  process. 


Latent  Semantic  Analysis  (LSA)  is  a  mathematical/statistical  technique  for  extracting  and 
representing  the  similarity  of  meaning  of  words  and  passages  by  analysis  of  large  bodies  of  text.  Laham, 
Darrell,  “LSA  website  Executive  Summary,”  Latent  Semantic  Analysis  @  CU  Boulder,  accessed  April  10, 
2014,  http;//lsa.colorado.edu/. 

^3  Thomas  K.  Landauer,  Peter  W.  Foltz,  and  Darrell  Laham,  “An  Introduction  to  Latent  Semantic 
Analysis,”  Discourse  Processes  25,  no.  2-3  (1998):  25 9-84. Singular  Value  Decomposition  (SVD)  is  a 
general  form  of  factor  analysis  to  condense  a  very  large  matrix  of  word-by-context  data  into  a  much 
smaller,  but  still  large-typically  100-500  dimensional -representation. 
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B.  SPECIFIC  RECOMMENDATIONS  TO  CDP  AND  WTTM 

The  CDP  has  been  effective  in  producing  equipment  for  warfighters;  however, 
this  process  still  contains  faults  that  contribute  to  developing  and  fielding  poorly  designed 
equipment  (e.g.,  ACU,  CROW,  THOR  III,  ITAS,  and  GROSS)  as  identified  in  the 
previous  chapter.  A  recommendation  to  help  the  CDP  mitigate  the  production  of  flawed 
equipment  to  soldiers  is  applying  sensemaking  of  narratives  as  a  feedback  mechanism. 
Collection  and  analysis  of  narratives  could  be  implemented  during  two  separate  steps  of 
Phase  4  of  the  CDP,  1)  initial  manufacturing,  and  2)  preliminary  equipment  fielding. 
During  the  initial  manufacturing  step,  a  small  sample  size  of  equipment  is  produced  for 
operational  testing,  which  is  the  first  time  when  warfighters  are  able  to  use  the  equipment 
in  non-scripted  but  controlled  scenarios  and  provide  feedback.  The  PMs  can  collect  and 
analyze  warfighter  narratives  as  an  operational  testing  feedback  mechanism  to  gain  their 
first  hand  experiences  with  equipment  performance.  Information  gained  could  assist  in 
initial  modifications  and  adjustments  prior  to  the  preliminary  fielding  of  equipment. 

During  preliminary  fielding,  a  small  baseline  of  equipment  is  issued  to  select  units 
for  additional  testing  in  a  training  or  combat  environment.  In  contrast  to  the  previous 
testing,  this  step  allows  for  warfighters  to  use  the  equipment  in  uncontrolled 
environments  and  provide  their  narratives.  More  importantly,  this  step  affords  PMs  and 
equipment  developers  an  opportunity  to  collect  and  analyze  soldier  narratives  based  on 
their  experiences  in  either  combat  or  training  conditions.  Once  again,  information  and 
insights  drawn  from  soldier  narratives  could  assist  in  modifications  and  adjustments 
before  mass  fielding. 

Similarly,  obtaining  and  analyzing  narratives  could  be  included  during  the 
ART/TSOA  events.  Following  soldiers’  testing  of  developmental  equipment,  TSOA 
evaluator  can  use  open-ended  prompts  to  elicit  warfighter  narratives  associated  with  their 
experiences  with  the  tested  equipment.  Obtaining  and  analyzing  narratives  can 
complement  the  WTTM  ART/TSOA  feedback  mechanisms  by  providing  additional 
insights. 
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C.  EXTENSION  OF  NARRATIVES  IN  SPECIAL  OPERATIONS  AND 
GEOTAGGING 

Conducting  special  operations  involves  more  than  just  operational  planning  and 
intelligence  analysis.  In  both  Unconventional  Warfare  (UW)  and  Foreign  Internal 
Defense  (FID)  operations,  Army  Special  Operations  Forces  (ARSOF)  soldiers  must  deal 
with  various  challenging  issues,  such  as  foreign  cultures  and  values.  It  is  possible  that 
collecting  and  analyzing  stories  from  indigenous  populations  could  yield  information 
about  their  culture  and  values  that  can  be  useful  for  soldiers  during  operations.  Similarly, 
collecting  and  analyzing  soldier  narratives  about  their  experiences  and  interactions  with 
foreign  cultures  could  also  provide  rich  data  and  insights.  Applying  this  idea,  ARSOF 
commanders  and  intelligence  specialists  could  use  open-ended  prompts  about  foreign 
cultures  to  collect  soldier  narratives  during  mission/operation  debriefs.  These  narratives 
associated  with  relevant  themes  and  signifiers  (e.g.,  the  population  was  friendly,  neutral, 
or  hostile)  could  be  analyzed  to  provide  useful  information,  especially  the  subtle  cultural 
idiosyncrasies.  Analysis  of  these  stories  can  aid  in  simplifying  complex  problems 
associated  with  UW  and  FID  operations,  which  can  lead  to  a  deeper  understanding  of 
foreign  cultures  and  populations. 

Obtaining  and  analyzing  both  native  population  and  soldier  narratives  to  gain 
information  on  cultures  can  be  expanded  even  further  through  geotagging^^  to  create  an 
overlay  as  part  of  a  geospatial  knowledge  repository.  Warfighters  can  create  narratives 
based  on  their  interactions  with  locals  and  geotag  them  in  a  manner  similar  to 
Lighthouse,^^  a  product  developed  by  the  Naval  Postgraduate  School’s  Common 
Operational  Research  Environment  (CORE)  Eab.  As  narratives  would  be  collected  about 
a  specific  culture  and/or  location,  then  aggregated  and  analyzed  to  possibly  provide 
insights  into  the  region’s  cultural  geography.  Additionally,  sensemaking  of  these 


An  electronic  tag  that  assigns  a  geographical  location  to  a  photograph  or  video,  a  posting  on  a  social 
media  website,  etc.  Oxford  Dictionary,  “Geotag:  Definition  of  Geotag  in  Oxford  Dictionary  (American 
English)  (US),”  Oxford  Dictionaries,  accessed  May  9,  2014,  http://www.oxforddictionaries.com/us/ 
definition/american_english/geotag. 

Lighthouse  is  a  program  designed  to  collect  socio-cultural  data  in  the  field  and  structure  it  for 
geospatial,  temporal,  or  social  network  analysis.  The  Lighthouse  Project,  “About  Lighthouse,”  Lighthouse, 
accessed  May  9,  2014,  http://lhproject.info/about-lighthouse/. 
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narratives’  associated  themes  and  signifiers  can  provide  context,  and  amplify  meanings  of 
service  members’  and  local  population’s  experiences.  Soldiers  can  access  this  geospatial 
knowledge  repository  to  better  understand  the  cultural  environment  of  a  region  prior  to 
their  arrival. 

The  similar  concept  of  applying  geotags  to  text  has  been  successfully  used  for 
various  applications,  such  as,  1)  using  story  maps  as  a  means  of  organizing  and 
presenting  information,^^  2)  gathering  information  through  crowd  sourcing  during  the 
Haiti  Earthquake  crisis,^^  3)  using  twitter  to  collect  and  share  near  real-time  information 
about  specific  locations  or  events  (e.g.,  Boston  Marathon  Bombers), and  4)  most 
commonly  providing  reviews  about  businesses  (erestaurants,  retail  stores,  etc.)  on 
websites,  such  as  Yelp,  TripAdvisor,  and  Google.  Although  the  concept  of  geotagging  is 
not  novel,  the  use  of  signified  narratives  in  conjunction  with  geotagging  to  capture 
cultural  geography  is  innovative. 


Story  maps  tell  the  story  of  a  place,  event,  issue,  trend,  or  pattern  in  a  geographic  context.  They 
combine  interactive  maps  with  other  rich  content  -  text,  photos,  video,  and  audio  -  within  user  experiences 
that  are  basic  and  intuitive.  Environmental  Systems  Research  Institute,  Inc.,  “What  Is  a  Story  Map?  I  Story 
Maps.”  http://storymaps.arcgis.com/en/articles/what-is-a-story-map/. 

Monica  Hesse,  “Crisis  Mapping  Brings  Online  Tool  to  Haitian  Disaster  Relief  Effort,”  The 
Washington  Post,  January  16,  2010,  sec.  Print  Edition,  http://www.washingtonpost.com/wp-dyn/content/ 
article/2010/01/15/AR2010011502650.html. 

Sharon  Myrtle  Parades!,  “Geotagging  Tweets  Using  Their  Content.,”  in  FLAIRS  Conference,  2011, 
http://www.aaai.org/ocs/index.php/FLAIRS/FLAlRSll/paper/viewFile/2617/3058. 
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APPENDIX  A.  INTEGRATED  DEFENSE  ACQUISITION,  TECHNOLOGY,  AND  LOGISTICS 

MANAGEMENT  LIFE  CYCLE 
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APPENDIX  B.  EXAMPLE  OF  THE  ART/TSOA  ASSESSMENT 
TOOL  (WTTM  LOGISTIC  FACTORS) 


FOUO  -  ART  TSOA  SYSTEM  ASSESSMENT  PROTOCOL 

WTTM  Lesiitic  Factors 

Factor  1;  Rtsporuo  to  Systtm  Malfunctions  (KCXB,  lUBIP:  C6  Non-Intrusiva] 

Required  Location  for 

Troubleahootinc/ltepoir 

Tools  Required  to 

TrouMedKwt/Repeir  System 

Skills  Required  to  Troubleshoot/Repeir  System 

Added  Risk  to  Soldier  Caused  By 
Performing  Troubleshootlng/Rep^ 

EL 

ML 

EH 

Doesn't  nnatter  -  system  self- 
troubleshoots  and  repairs  without 
network  connection 

None 

None 

None 

10 

10 

10 

Doesn’t  matter  -  system  self- 
troubleshoots  and  repairs  with  network 

None 

Common  soldier  skills 

Slight  risk  of  minor  injury 

9 

9 

9 

Doesn't  matter  -  user  can 

troubleshoot/repair  without  network 

Multiple  tools  available  in  unit 

User  can  acquire  with  limited  system  use 

Significant  risk  of  minor  injury 

8 

8 

8 

Doesn't  matter  -  user  can  T/R  with 
network  connection/communications 

Small  number  of  tools  available 

in  unit 

User  can  acquire  with  significant  system  use 

7 

7 

7 

On-site  only  without  network 
connection/communications 

Multiple  tools  supplied  with 
system 

User  can  acquire  via  soldier  train-the-trainer 

Slight  risk  of  a  major  injury 

6 

6 

6 

On-site  only  with  network 
connection/communications 

Small  number  of  tools  supplied 
with  system 

Skills  can  be  trained  on  in  a  short  period  of  time 
by  FSR 

5 

5 

Must  be  done  off-site  at  next  higher  unit's 
location 

Multiple  tools  supplied  with 
system 

Skills  available  to  unit  only  through  significant 
training  by  FSR 

Significant  risk  of  a  major  injury 

4 

4 

4 

Off-site  beyond  next  higher  unit's  location 

Specialized  tools  available  In 
Army  logistic  system 

User  can  acquire  only  through  significant 
training  in  CONUS 

3 

3 

3 

In-theater  Contractor  Support  location 

Specialized  tools  available  only 
through  commercial  order 

Skills  acquired  only  through  formal  civilian 
education 

Slight  risk  to  life 

2 

2 

2 

Must  be  sent  back  to  CONUS  location 

Specialized  tools  not  available 
to  unit 

Must  work  for  the  system  vendor  to  know  how 
to  troubleshoot  &  repair 

Significant  risk  to  life 

1 

1 

Factor  2:  Rounnt  navamaoMMalmtnanea  (Tndudng  software  updatas/upgradas)  [lUBiP: 

a  PersMent.  04  Sustainebie) 

Frequency 

System  Downtime 

Skills  Required  for  PM 

Notes  &  Comments 

EL 

ML 

EH 

None  required 

0 

None 

This  factor  is  only  addressing  pre- 
planned/known  preventative 
maintenan<£, 

10 

10 

10 

None  required  during  mission  use  at  the 
Combat  Outpost 

less  than  1  hour 

Common  soldier  skills 

9 

9 

9 

1  -  2  hours 

Skills  can  be  self-taught  with  limited  system  use 

8 

8 

8 

2  -4  hours 

Skills  can  be  seit-taught  with  significant  system 

7 

7 

7 

Some  amount  required  during  a  mission  - 
no  detrimental  mission  impact 

4  -  6  hours 

Skills  acquired  via  soldier  train-the-trainer 

6 

6 

6 

6  - 12  hours 

Skills  can  be  trained  on  in  a  short  period  of  time 
only  by  FSR 

5 

5 

5 

12  •  24  hours 

Skills  can  be  trained  on  only  through  significant 
training  by  FSR 

4 

4 

4 

Significant  amount  required  during 
mission  -  will  degrade  mission 
performance 

1-2  days 

Skills  available  to  unit  only  through  significant 
trainine  in  CONUS 

3 

3 

3 

3 -5  days 

Skills  can't  effectively  be  trained,  they  can  be 
acquired  onlv  throueh  formal  civilian  education 

2 

2 

2 

more  than  a  week 

Skills  unique  to  vendor  engineering  group 

1 

1 
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Factor  3:  Supply  Support  Plannint  Impact  (ICD<8) 

Consumables  Required  by  System 

Impact  on  Plani^ 

Notes  &  Comments 

EL 

ML 

EH 

No  consumables  required  by  the  system 

No  noticible  impact  to  normal  logistics  planning 

Examples  of  consumables  are  fuel,  filters, 
oil. 

10 

10 

10 

All  consumables  are  supplied  in  a  normal 
LOGPAC  ordered  by  COP 

Minor  advanced  planning  required 

9 

9 

9 

8 

8 

8 

7 

7 

7 

System  requires  consumables  that  add 
additional  volume  to  routine  LOGPAC 
levels 

Typical  advanced  planning  required 

6 

6 

6 

5 

5 

5 

4 

4 

4 

System  requires  consumables  outside  of 
the  normal  LOGPAC 

Significant  advanced  planning  required 

3 

3 

3 

2 

2 

2 

Unique  consumable  requirements  not 
available  in  normal  LOGPAC 

Support  requirements  significantly  stress  normal  logistics  planning  with  unique 
reauirements 

1 

1 

1 

Factor  4:  Rapalr  k  Raplacamartt  Parts  (KD^J 

Number  of  Parts  Requiring  Repair  or 
Replacement 

Potential  Response  Time  for 
On-demand  Repair  Parts 

Acquisition  Process  Required  for  Obtaining 
Parts 

Notes  &  Comments 

EL 

ML 

EH 

None 

On-hand 

System  automatically  orders  and  obtains  parts 
prior  to  end  user  identifying  need 

10 

10 

10 

Very  small  number 

Less  than  1  day 

Normal  unit  infrastructure  sufficient  to  obtain 

oarts 

9 

9 

9 

Additional  unit  resources  needed  to  obtain  parts 

8 

8 

8 

7 

7 

7 

Small  number 

1  to  3  days 

6 

6 

6 

3  -  5  days 

5 

5 

5 

5-7  days 

4 

4 

4 

Large  number 

1  -2  weeks 

3 

3 

3 

2 

2 

2 

2  weeks  or  greater 

Non-military,  proprietary  ordering  with 
commercial  shipping 

1 

1 

1 

Factor  5:  Setup  Requirements  On-site  (iC[>-CB) 

Impact  of  Setup  on  COP  Operations 

Tools  and  Equipment  Required 
for  Setup 

Skills  Required  to  Setup  System 

Notes  &  Comments 

EL 

ML 

EH 

Setup  process  enhances  COP  operations 
and  security 

None  required 

None 

10 

10 

10 

None 

Unit  equipment  sufficient  for 
system  setup 

Common  soldier  skills 

9 

9 

9 

Skills  can  be  self-taught  with  limited  system  use 

8 

8 

8 

Setup  inflicts  minor  degradation  on  COP 
ooerations 

Skills  can  be  self-taught  with  significant  system 

7 

7 

7 

Setup  degrades  COP  operations 

Non-unit  equipment  and  tools 
needed 

Skills  acquired  via  soldier  train-the-trainer 

6 

6 

6 

Skills  can  be  trained  on  in  a  short  period  of  time 
onlv  bv  FSR 

5 

5 

5 

Skills  can  be  trained  on  only  through  significant 
training  bv  FSR 

4 

4 

4 

Setup  negatively  impacts  COP  security 

Special,  difficult  to  obtain  tools 
and  equipment  needed 

Skills  available  to  unit  only  through  significant 
training  in  CONUS 

3 

3 

3 

Skills  can't  effectively  be  trained,  they  can  be 
acquired  only  through  formal  civilian  education 

2 

2 

2 

Requirements  enhance  enemy  ability  to 
engage  COP 

Skills  unique  to  vendor  engineering  group 

1 

1 

1 

Factor  b:  System  Transportation  Requirements  (lUBiP:  C5  scalable) 

Transportation  Impact  on  COP 

Imposed  planning  requirements 

Notes  &  Comments 

EL 

ML 

EH 

COP  operations  are  enhanced  by  the 
□resence  of  system  transoortation 

No  noticible  impact  to  normal  transportation  planning 

Requirements  include  handling,  HAZMAT 
and  other  non-standard  transportation 
elements 

10 

10 

10 

None 

Minor  advanced  planning  required 

9 

9 

9 

8 

8 

8 

Presence  ot  system  transportation  assets 
has  a  minor  negative  effect  on  COP 

7 

7 

7 

System  transportation  assets  has  a 
signficant  negative  impact  on  COP 
ooerations 

Typical  advanced  planning  required 

6 

6 

6 

5 

5 

5 

4 

4 

4 

Transportation  assets  has  a  significant 
negative  impact  on  COP  security 

Significant  advanced  planning  required 

3 

3 

3 

2 

2 

2 

Transportation  assets  has  a  positive  effect 

on  adversary's  TTPs 

Support  requirements  significantly  stress  normal  transportation  planning  with 

unioue  reauirements 

1 

1 

1 

DtSTRlBimON  STATEMENT  E.  Distrlbutton  authorized  to  DoD  components  only  due  to  the  proprietary  nature  or  assessment  data.  23  April  2013.  Other  requests  sha 
_ ART  TSQA  Core  Group  at  ARLUMC.TainDa.FL _ 

be  referred  to  DFP 
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APPENDIX  C.  EXAMPLE  OF  HIERARCHICAL  TASK 
ASSESSMENT  FOR  CENTURION  MULTI-MISSION  SYSTEM 

(CMMS) 


CMMS  Hierarchical  Task  Analysis 

The  following  pages  contain  the  scenario  used  to  assess  hierarchical  tasks  on  the 
Centurion  System.  We  assessed  two  experts  from  Night  Vision  Labs  on  these  tasks. 
Afterward,  evaluator  used  30  minutes  to  train  two  soldiers  on  the  set-up  and  operation  of 
the  entire  system,  including  the  interface.  Afterward,  these  two  were  subjected  to  the 
identical  scenario  as  the  two  experts.  After  that,  we  took  two  complete  novices  and 
assessed  their  performance  on  the  scenario. 

Set  up  and  make  operational  the  trailer-mounted  system 

1 .  Set  up  local  security  and  make  operational  the  C2  back  pack  system. 

Operate  the  Centurion  System 
A.  Identify  Targets  Using  the  Ranger  R3D  Radar 

1 .  Ensure  the  Radar  is  Active  and  is  in  Doppler  Short  Mode 

a.  Locate  the  R3D  Radar  (STS- 1400)  menu  in  the  tree  view  (left  side 
of  the  screen  displaying  all  of  the  sensors  available  to  each  system) 
and  ensure  the  colored  rectangle  next  to  it  does  not  have  a  down 
arrow  next  to  it. 

b.  Select  the  R3D  Radar  (STS  1400)  sensor  form  view  in  the  tree 
view  by  left  clicking  on  the  word  “R3D  Radar” 

c.  Locate  the  Properties  button  on  the  simple  form  view  (simple  form 
view  appears  to  the  right  of  the  map  once  “R3D  Radar  (STS- 
1400)”  is  selected) 

d.  Select  the  Properties  button  in  the  R3D  Radar  (STS- 1400)  simple 
form  view  by  left  clicking  on  the  button  Properties 

e.  Locate  the  Mode  drop  down  menu  in  the  R3D  Radar  (STS- 1400) 
“Properties”  window  and  ensure  it  is  in  Active  mode.  If  not,  open 
the  drop  down  menu  and  select  Active. 

f.  Locate  the  Doppler  heading  at  the  lower  left  corner  of  the 
“Properties”  window  and  ensure  the  Scan  Mode  is  set  to  Doppler. 

If  not,  open  the  drop  down  menu  and  select  Doppler. 

g.  In  the  Time  On  Target  Drop  Down  menu  below  the  Scan  Mode, 
ensure  the  selection  is  on  Short. 
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h.  Locate  the  OK  button  at  the  bottom  of  the  “Properties”  menu 

i.  Select  the  OK  button  to  close  out  of  the  R3D  Radar  (STS- 1400) 
“Properties”  window  and  to  accept  any  changes  made  to  the  R3D 
Radar  modes. 

B.  Classify  the  Radar  Target  (Vehicle?  Personnel?) 

2.  Open  C16  Video  window  by  clicking  on  the  C16  Video  level  (beneath 
C16  Imaging  Suite  level)  in  the  tree  view. 

Double-click  in  the  C16  Video  window  to  make  it  “Active,”  which  is  indicated  by 
a  green  border  on  the  window. 

Option  1 :  Through  “Slew  to  Click”  mode 

a.  Activate  the  “Slew  to  Click”  mode  by  locating  the  “Slew  to  Click” 
icon  (crosshairs)  in  the  toolbar  at  the  top  of  the  screen  or  using  the 
hotkey  “CTRL-i-S” 

b.  Click  on  the  “current”  R3D  radar  hits,  i.e.,  largest  radar  hits  on  the 
screen  (dots  on  the  map  matching  the  color  of  the  larger  radar  scan 
sector  on  the  map)  to  slew  camera  to  potential  targets. 

c.  Identify  current  targets  as  vehicles  with  C16  video. 

Option  2:  Manually  move  C16  camera  with  gamepad  joystick 

d.  Ensuring  the  C16  video  window  is  green,  move  left  bottom 
joystick  on  the  gamepad  joystick  until  the  pink  camera  fans  on  the 
map  are  aligned  with  the  potential  radar  hits  on  the  map,  adjust  as 
necessary. 

e.  Identify  targets  as  vehicles  with  C16  video. 

C.  Investigate  the  target  using  camera  zoom 

1 .  Press  the  up  and  down  directional  buttons  on  the  upper  left  portion  of  the 
gamepad  joystick  to  zoom  in  or  out  and  assess  vehicle  targets  with  the 
C16  video. 

a.  Locate  the  grid  coordinates  of  the  target 

2.  Mouse  over  map 

a.  Identify  the  pink  camera  fans  on  the  map  to  indicate  where  the 
cameras  are  looking 

b.  Once  the  camera  fans  line  up  with  the  radar  targets  of  interest, 
mouse  over  the  targets  to  obtain  the  grid  coordinates,  which  are 
displayed  at  the  bottom  right  of  the  screen 

c.  Pass  target  information  (grid,  description)  to  the  TOC  using 
software 
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Training  Results: 


Setup  Type: 

Expertise' 

Time  (min:sec): 

Evaluator  Comments: 

Trailer 

Expert 

Remote  Security  set-up 

Expert 

Trailer 

Trained  by  Expert 

Remote  Security  set-up 

Trained  by  Expert 

Trailer 

Novice 

Remote  Security  set-up 

Novice 
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APPENDIX  D.  POWERPOINT  COLLECTION  TOOL 
ADMINISTERED  TO  GATHER  SERVICE  MEMBER  NARRATIVES 


Newly  Fielded  Equipment  Soldier-user  Survey 

The  narrative  story  and  the  answers  to  questions  pertaining  to  that  narrative  wiii  be  used  as  part  of  a  research  project 
being  conducted  by  the  Navai  Postgraduate  Schooi.  Narrative  anaiysis  wiii  serve  to  inform  and  improve  the  process  by 
which  new  equipment  is  fieided,  particuiariy  in  the  areas  pertaining  to  the  user,  iogistics,  and  technoiogicai  aspects  of  the 
new  equipment. 

iAW  institutionai  Review  Board  (iRB)  Reguiations  we  need  your  consent  based  on  the  foiiowing  pertinent  information: 

•  Providing  your  narrative  is  VOLUNTARY,  and  refusal  to  participate  will  involve  no  penalty.  The  soldier  may  discontinue 
participation  at  any  time  throughout  the  survey  process  without  penalty. 

•  Estimated  time  to  complete  the  survey  is  30-60  minutes  (time  needed  to  enter  the  narrative  and  answer  a  few  questions). 

•  ALL  narratives  are  anonymous  and  will  not  be  associated  to  the  soldier's  name,  SSN,  or  unit 

•  ALL  information  collected  will  be  saved  on  a  NPS  secure  server  and  provided  to  the  NPS  IRB  for  long  term  storage  at  completion 
of  the  research. 

•  There  are  NO  costs  to  the  subject  that  may  result  from  participation  in  the  research. 

•  The  benefits  of  the  research  will  be  found  in  improving  the  rapid  fielding  process.  You  will  not  directly  benefit  from  participating. 

•  There  are  no  foreseeable  risks  associated  with  participating. 

•  The  survey  consists  of  two  parts:  1)  entering  your  narrative  2)  Answering  questions  related  to  the  narrative  to  better  clarify  the 
details  of  the  interactions  with  the  new  equipment. 

If  you  have  any  questions  about  the  research  or  you  experience  an  injury,  please  contact  the  Principal  Investigator,  Dr. 
Michael  Jay^  831 .656.2536,  miiaveSbnos.edu.  Questions  about  your  rights  as  a  research  subject  or  any  other  concerns 
may  be  addressed  to  the  Navy  Postgraduate  School  IRB  Chair,  Dr.  Larry  Shattuck,  831 .656.2473,  iashattuSbnos.edu. 


IRB 
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Newly  Fielded  Equipment  Soldier-user  Survey 

Telling  stories  is  an  essential  part  of  being  human  -  it  is  how  we  pass  along  culture, 
traditions,  ideas,  and  experiences.  With  that  in  mind,  as  part  of  improving  the 
process  of  fielding  new  equipment  the  Adaptive  Red  Team  of  Deployable  Force 
Protection  seeks  your  input:  rather  than  asking  you  a  series  of  questions,  we  are 
asking  you  to  tell  a  story  about  your  experience  with  newly  fielded  equipment.  After 
you  have  told  your  story,  you  will  be  given  a  chance  to  amplify  it  through  a  series  of 
questions. 

We  do  not  seek  any  personal  information  about  you.  We  are  only  interested  in  your 
story. 

On  the  next  page  is  blank  box  for  you  to  use  to  tell  us  the  newly  fielded  equipment 
you  had  experience  with  while  deployed.  Please  type  in  the  equipment's  name  and 
then  move  to  the  next  screen. 


Narrative  prompt 
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Newly  Fielded  Equipment  Soldier-user  Survey 


The  new  equipment  in  your  story  improved... 


Click  and  drag  the 
bubble  inside  the 
triangle  to  a  position 
that  best  describes 
your  answer  relative 
to  the  story  you  have 
just  told. 


Reduced  demands  on  your  time 


Security 


Unit  morale 


This  is  an  example  Triad  with  the  theme  “improved” 
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Newly  Fielded  Equipment  Soldier-user  Survey 


Click  and  drag  the  bubble  inside  the  scale  to  a  position  that  best  describes  your  answer  relative  to  the  story  you  have  just  told. 


In  your  story,  the  new  equipment  impressed  you  most  favorably  by... 


The  value  added  to 
the  unit's  capabilities 


The  impact  it  had  on 
many  other  facets  in 
the  unit 


In  your  story,  the  new  equipment  disappointed  you  most  by... 


The  impact  it 
had  on  the  unit 


Its  inability  to  deliver 
on  expectations 


This  is  an  example  of  two  Dyads  with  the  themes  “impressed”  and  “disappointed” 

Multiple  Choice  questionnaire 
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APPENDIX  E.  EIGHT  TRIAD  THEMES,  PROMPTS,  AND 

SIGNIFIERS 


Theme:  Improved 

Prompt:  “The  new  equipment  in  your  story  improved. . 

Signifiers:  (a)  Reduced  demand  on  your  time;  (b)  Security;  or  (c)  Unit  Morale 

Theme:  Failed 

Prompt:  “The  new  equipment  in  your  sioxy  failed  to  improve. . 

Signifiers:(a)  Reduced  demand  on  your  time;  (b)  Security;  or  (c)  Unit  Morale 

Theme:  Benefit 

Prompt:  “The  benefit  to  your  unit  provided  by  the  new  equipment  was  in. . .” 
Signifiers:  (a)  Basic  needs;  (b)  Quality  of  Life;  (c)  Safety  and  security 

Theme:  Harm 

Prompt:  “The  harm  to  your  unit  caused  by  the  new  equipment  was  in. . .” 

Signifiers:  (a)  Basic  needs;  (b)  Quality  of  Life;  (c)  Safety  and  security 

Theme:  Aspects  Improved 

Prompt:  “Based  on  your  story,  what  aspect  of  the  new  equipment  could  be  improved 
the  most?” 

Signifiers:  (a)  Soldier  training  requirements;  (b)  Logistic  requirement  (maintenance, 
parts,  support,  etc);  (c)  Equipment  combat  performance 

Theme:  Greatest  Value 

Prompt:  “In  your  story,  the  new  equipment’s  greatest  value  is  found  in. . .” 

Signifiers:  (a)  Enhanced  performance  obtained  by  integrating  with  other  systems;  (b) 
The  impact  the  new  equipment  had  on  the  unit’s  combat  capabilities;  (c)  The  impact 
it  had  on  logistical  requirements  (parts,  maintenance,  etc.)  relative  to  capability 
gained 
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Theme:  Greatest  shortcomings 

Prompt:  “In  your  story,  the  new  equipment’s  greatest  shortcoming  is  found  in. . 

Signifiers:  (a)  Enhanced  performance  obtained  by  integrating  with  other  systems;  (b) 
The  impact  the  new  equipment  had  on  the  unit’s  combat  capabilities;  (c)  The  impact 
it  had  on  logistical  requirements  (parts,  maintenance,  etc.)  relative  to  capability 
gained 


Theme:  Attention 

Prompt:  “The  situation  I  recalled  in  the  story  should  get  the  attention  of . . .” 

Signifiers:  (a)  Congress;  (b)  The  media;  (c)  The  next  unit  that  will  field  the 
equipment 
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APPENDIX  F.  TEN  DYAD  THEMES,  PROMPTS,  AND  SIGNIFIERS 


Theme:  Impressed 

Prompt:  “In  your  story,  the  new  equipment  impressed  you  most  favorably  by. . 

Signifiers:  (a)  The  value  added  to  the  unit’s  capabilities;  (b)  The  impact  it  had  on 
many  other  facets  in  the  unit 

Theme:  Disappointed 

Prompt:  “In  your  story,  the  new  equipment  disappointed  you  most  by. . .” 

Signifiers:  (a)  The  impact  it  had  on  the  unit;  (b)  Its  inability  to  deliver  on 
expectations 

Theme:  Approach 

Prompt:  “In  your  story,  your  approach  to  using  the  new  equipment  was. . .” 
Signifiers:  (a)  Cautious;  (b)  Enthusiastic 


Theme:  Revolutionary 

Prompt:  “Based  on  the  experience  you’ve  related,  how  revolutionary  was  the  new 
equipment?” 

Signifiers:  (a)  So  revolutionary  that  it  transformed  the  unit’s  capabilities;  (b)  So 
revolutionary  that  it  was  too  hard  to  understand,  use,  or  implement 


Theme:  Troubleshooting  and  Repairing 

Prompt:  “The  effort  you  and  your  unit  exerted  in  troubleshooting  and  repairing  the 
new  equipment  was. . . ” 

Signifiers:  (a)  Insignificant;  (b)  Burdensome 


Theme:  Implement  and  Operate 

Prompt:  “The  skills  necessary  for  the  user  to  correctly  implement  and  operate  the 
new  equipment  in  your  story  were. . .” 

Signifiers:  (a)  Easy  for  anyone;  (b)  Difficult  for  anyone  except  trained  specialists 
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Theme:  Maintain 

Prompt:  “The  skills  necessary  to  maintain  the  new  equipment  in  your  story  were. . . 

Signifiers:  (a)  Simple  -  nearly  anyone  could  maintain  the  new  equipment’  (b) 
Difficult  -  special  training  was  needed  because  the  required  maintenance  skills  were 
specialized  or  difficult 


Theme:  Operability  and  Survivability 

Prompt:  “The  operability  and  survivability  of  the  new  equipment  in  your  story 
were...” 

Signifiers:  (a)  So  good  that  you  and  the  unit  could  always  rely  on  it;  (b)  So 
burdensome  or  fragile  that  no  one  could  count  on  it 


Theme:  Integrate 

Prompt:  “How  easy  was  it  for  you  and  your  unit  to  integrate  the  new  equipment  with 
other  combat  systems  and  the  operational”  environment? 

Signifiers:  (a)  It  integrated  so  easily  that  its  value  added  was  easy  to  notice;  (b)  It  was 
so  difficult  to  integrate  that  we  gave  up  on  it 


Theme:  Impact 

Prompt:  “The  new  equipment  in  your  story  had  a  significant  impact  on. . .” 
Signifiers:  (a)  Logistic  and  maintenance  demands;  (b)  Successful  operations 
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APPENDIX  G.  EQUIPMENT  LIST  ACCORDING  TO  PRIMARY 
FUNCTION  OF  SHOOT,  MOVE,  COMMUNICATE,  AND  SURVIVE 
ACCORDING  TO  FIELD  MANUAL  7-2I.I3,  THE  SOLDIER’S 

GUIDE 

Shoot 

Common  Remotely  Operated  Weapon  Station  (CROW) 

M3 20  Grenade  Launcher  Module 

Mark  VII  Handheld  Eyesafe  Laser  Rangefinder  (MK  7E) 

M240L  Machine  Gun 

M249  Squad  Automatic  Weapon  (SAW) 

M2A1  .50  Caliber  Machine  Gun 
MK  48  Machine  Gun 
PGM  Mortar  Round 

Improved  Target  Acquisition  System  (ITAS) 

M-4  Carbine 

XM  2010  Enhanced  Sniper  Rifle 

AN/PVS  6  Mini  Eyesafe  Laser  Infrared  (MELIOS) 

Lightweight  Handheld  Mortar  Ballistic  Computer  (LHMBC) 

M240  Machine  Gun  Tripod 

AN/PVQ  20  Dual  Sensor  Night  Vision  Goggle  (DSNVG) 

M-4  Carbine  Eree  Eloating  Barrel 
Mortar  Eire  Control  System  (MECS) 

Night  Vision  Goggles 

AN/PEQ-15  Advanced  Target  Pointer  Illuminator  Aiming  Laser  (ATPIAL) 
Stoner  Rifle-25  (SR-25) 

XMllO  Semi-Automatic  Sniper  Rifle  (SASS) 

Move 

Medium  Tactical  Vehicle  (MATV) 

RG-33  (Multi-purpose  Mine-resistant  Ambush  protected  Infantry  Mobility 
Vehicle) 

Light  Medium  Tactical  Vehicle  (LMTV) 
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Mine  Resistant  Ambush  Protected  (MRAP) 

MaxxPro  Recovery  Vehicle  (MRV) 

RG  3 1  (Multi-purpose  Mine-resistant  Ambush  protected  Infantry  Mobility 
Vehicle) 

Remote  Operating  Tactical  Vehicle 
Double  V-Hull  Stryker 

Communicate 

Secure  Electronic  Enrollment  Kit  (SEEK  II) 

AN/PRC-I I7G  Multiband  Manpack  Radio 

RT-1523E  Single  Channel  Ground  and  Airborne  Radio  System-Advanced  System 
Improvement  Program  (AS  IP) 

Eorce  XXI  Battle  Command  Brigade  and  Below  (PBCB2) 

Soldier  Worn  Integrated  Power  Equipment  System  (SWIPES  Kit) 

Satellite  Communication  Radio  (SATCOM) 

Biometrics  Automated  Toolset  System  (BATS) 

AN/PSN-13  Defense  Advanced  GPS  Receiver  (DAGR) 

Distributed  Tactical  Communication  System  (DTCS) 

EA-lOu  Handheld  Easer  Marker  (HME) 

Remote  Operational  Video  Enhanced  Receiver  (ROVER) 

AN/PYQ-10  Simple  Key  Eoader  (SKE) 

Garmin  Wrist  Global  Positioning  System  (GPS) 

Survive 

Medium  Modular  Eightweight  Eoad-carrying  Equipment  -  Ruck  Sack  (MOEEE) 
Earge  Modular  Eightweight  Eoad-carrying  Equipment  -  Ruck  Sack  (MOEEE) 
Ground  Bass  Operational  Surveillance  System  (GBOSS) 

Centurion  Multi-Mission  System  (CMMS) 

Eong  Range  Thermal  Video  Imaging  System  (ERTV) 

M240E  Machine  Gun  Barrel  Bag 

Alice  Ruck  Sack 

Boots 

Eorward  Repair  System  (ERS) 

Air  Items 


74 


Assault  Kitchen 
AMPS  Generator 

Computer  Numerical  Control  Cutter  (CNC) 

Fueler 

CP  Medium  Tent 

Ml  120  HEMTT  Load  Handling  System  (LHS) 

MEP  321 A  Generator 
Automated  Systems  Scanner 
Palletized  Load  System  (PLS) 

Standard  Army  Maintenance  System-Enhanced  (SAMS-E) 

Standard  Automotive  Tool  Set  (SATS) 

Plate  Carrier 

Multi-Cam  Camouflage  /  CRYE 

THOR  III  (Man-portable  Counter  Radio-Controlled  Improvised  Explosive  Device 
and  Electronic  Warfare  system) 

Army  Combat  Uniform  (ACU) 

Improved  Outer  Tactical  Vest  (lOTV) 

Advanced  Combat  Helmet  (ACH) 

CREW  Duke  (Counter  Radio-controlled  Electronic  Warfare  jamming  system) 

Wolfhound  Handheld  Threat  Warning  System 

Boomerang  3  Shooter  Detection  System 

Bradley  Reactive  Armor 

M9  Aid  Bag 

Eield  Protect  Mask  (ProMask) 

AN/PSS-14  Handheld  Standoff  Mine  Detection  System 
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